System for audio distribution including network microphones for voice applications

ABSTRACT

Disclosed is a microphone system, comprising: one or more microphones each of which is adapted to receive acoustic audio signals and convert the same from acoustical energy signals into electrical audio signals and output the same as microphone output audio signals; and an acoustic echo cancellation (AEC) device adapted to receive the microphone output audio signals, and wherein the AEC device also receives as a second input a reference audio signal from a network, and wherein the AEC device is further adapted to cancel substantially all of the reference audio signal from the microphone output audio signals and output the same as a corrected audio signal, and wherein the reference audio signal comprises an audio signal generated by an external audio system.

PRIORITY INFORMATION

The present application claims priority under 35 U.S.C. § 120 as aContinuation-in-Part Applications to U.S. Non-provisional patentapplication Ser. No. 15/261,296, filed 9 Sep. 2016; U.S. Non-provisionalpatent application Ser. No. 15/261,296, claims priority under 35 U.S.C.§ 119(e) to U.S. Provisional Patent Application No. 62/216,971, filed 10Sep. 2015; U.S. Non-provisional patent application Ser. No. 15/261,296,claims priority as a Continuation-in-Part Applications to U.S.Non-provisional patent application Ser. No. 14/850,904, filed 10 Sep.2015; and U.S. Non-provisional patent application Ser. No. 14/850,904,claims priority under 35 U.S.C. § 119(e) to U.S. Provisional PatentApplication No. 62/048,722, filed 10 Sep., 2014, the entire contents ofall of which are expressly incorporated herein by reference.

CROSS REFERENCE TO RELATED APPLICATIONS

Related subject matter is disclosed in Applicants' and co-filed U.S.Non-provisional patent application Ser. No. 16/207,924, entitled “Systemfor Audio Distribution Including Network Microphones for VoiceApplications,” co-filed on Dec. 3, 2018, the entire contents of whichare expressly incorporated herein by reference. Related subject matteris disclosed in Applicants' U.S. Non-Provisional patent application Ser.No. 14/850,904, entitled “Improvements in Configuring a Control System,”filed on 10 September, 2015, the entire contents of which are expresslyincorporated herein by reference. Related subject matter is alsodisclosed in Applicants' U.S. Non-Provisional patent application Ser.No. 15/261,296, entitled “Acoustic Sensory Network,” filed on 9September, 2016, the entire contents of which are expressly incorporatedherein by reference.

BACKGROUND Technical Field

Aspects of the embodiments relate generally to control networks, andmore specifically to systems, methods, and modes for controllingcontrollable devices in the control network based on audio commandsalone, according to an aspect of the embodiments, and in further aspectsof the embodiments, controlling the controllable devices of the controlnetwork based on audio commands and other sensory information.

Background Art

Today, there are home control systems that include lighting, shades,environmental controls, security, audio-visual (AV), and other types ofsub-systems. In many of the currently available home control systems,the user can turn and off components of such systems (from hereon inreferred to as “controllable devices”), for example, lighting products,by a switch, one or more remote control (RC) devices (such as adedicated RC device, or through some other type of RC device), remotelythrough network messages (e.g., command and control messages through theinternet), and other means, such as speech.

In the latter case, users always desire more convenient methods formanaging controllable devices, and the advent of speech based control inhandheld devices has led to a desire for speech based control ofcontrollable devices. Currently, lights can be turned on and offautomatically when a user enters a room via use of a motion sensor.However, the motion sensor can take several minutes after an occupanthas left a room to turn off the lights. This leads to wasted energy andfrustration. Speech based control can allow a user to (relatively)quickly turn off lights while leaving a room. That is, when speech basedcontrol systems work.

As those of skill in the art can appreciate, there are several problemswith speech based control systems that must be addressed. Among them areinoperativeness, false positives, collocation issues, and privacyissues. In regard to inoperativeness, this is defined by the speechbased control system simply failing to respond at all to a propercommand. A false positive is when a user does not intend thecontrollable device to turn off, but it does. This can occur because thespeech recognition system misinterprets the recorded audible signal, andincorrectly applies a control, when one was not intended. Collocationissues can arise when two control devices are relatively close to eachother, and a command is heard by both, and both or the wrong device actson the command that was intended for a first control device, but not thesecond. The privacy issue arises when certain private areas of a home(e.g., the lavatory) are adjacent to other rooms; commands issued in ornear that room can be misinterpreted by the system, causing occupants tomake possible embarrassing counter-commands.

Accordingly, a need has arisen for more precise audible control of acontrol network and the controllable devices that make up the controlnetwork by providing systems, methods, and modes for controllingcontrollable devices in the control network based on audio commandsalone, according to an aspect of the embodiments, and in further aspectsof the embodiments, controlling the controllable devices of the controlnetwork based on audio commands and other sensory information.

Existing voice recognition systems (VRSs) are only capable of hearingvoices with a relatively high signal-to-noise ratio (SNR) in thepresence of multimedia audio (MMA) if the VRS is also the source of themultimedia audio—that is, if the VRS broadcasts MMA, as well as thevoice responses, then the VRS can effectively perform voice recognitionon the unknown spoken words. Nonetheless, voice recognition systems haveimproved over the years and cloud-based implementations (e.g., thosesystems that interconnect via the Internet to one or more remotely basedservers), have become popular in phones and home audio devices. Audiodevices like the Amazon Echo®, Google-Home®, and Apple HomePod® are someof the most popular audio playing products sold. These devices have theability to always listen and recognize keywords (i.e., they are always“on” or enabled, and substantially continuously monitor the output of amicrophone for keywords). Once a keyword is detected, the device recordsthe subsequent audio and sends it to the cloud for recognition (i.e.,remotely located server or serves), parsing and resulting action. Manyof these devices are also audio player devices and have speakersincorporated into them. Some devices can have auxiliary audio outputsthat allow for external amplifiers and speakers to be used for thecontent output from the device. These devices can also have one or moremicrophones used to listen for voice commands.

VRSs also use one or more of several signal processing techniques toimprove the SNR detection capability of the microphones. Many of thesetechniques involve the use of multiple microphones. These includealgorithms such as auto-mixing, beam forming, de-reverberation, noisesuppression, and gain control, among others. In addition, devices thatplay audio utilize acoustic echo cancellation (AEC) to help eliminatethe media sound played by the device from the microphone signal (i.e.,the electrical output signal of the microphone transducer). It is knownby those of skill in the art that an AEC implementation in a VRS cannoteliminate the sound from another multimedia device, such as a televisionplaying in the room, because the VRS that is using the AEC does not havea reference signal for the audio output by that device's speakers (i.e.,the speakers from the television).

Thus, a need exists to improve voice recognition systems such that theeffects of external audio, substantially regardless of its source, canbe compensated when processing spoken commands to the voice recognitionsystem.

SUMMARY

It is an object of the embodiments to substantially solve at least theproblems and/or disadvantages discussed above, and to provide at leastone or more of the advantages described below.

It is therefore a general aspect of the embodiments to provide systems,methods, and modes that will obviate or minimize problems of the typepreviously described by controlling controllable devices in a controlnetwork based on audio commands alone, according to an aspect of theembodiments, and in further aspects of the embodiments, controlling thecontrollable devices of the control network based on audio commands andother sensory information.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Further features and advantages of the aspects of the embodiments, aswell as the structure and operation of the various embodiments, aredescribed in detail below with reference to the accompanying drawings.It is noted that the aspects of the embodiments are not limited to thespecific embodiments described herein. Such embodiments are presentedherein for illustrative purposes only. Additional embodiments will beapparent to persons skilled in the relevant art(s) based on theteachings contained herein.

According to a first aspect of the embodiments, a system is provided fordetermining which controllable device an audible command is directedtowards, the system comprising two or more controllable devices; two ormore electronic devices, each of which is adapted to receive the audiblecommand, add a respective electronic device identifier to the receivedaudible command, time and date stamp the received audible command, andtransmit the respective time and date stamped versions of the audiblecommand, and wherein each of the two or more electronic device arefurther adapted to control respective ones of the two or morecontrollable devices; and a central processor adapted to receive each ofthe transmitted time and date stamped versions of the audible commandand perform processing based on the time and date stamp, wherein theelectronic device that reports the earlier time and date stamp, asascertained by the respective electronic device identifier and acomparison of each of the time and date stamps performed by the centralprocessor, is the electronic device to which the audible command isdirected towards.

According to the first aspect of the embodiments, the time and datestamped versions of the audible command further conveys amplitudeinformation of the respectively received audible command, and whereineach of the two or more electronic devices are further adapted tocompare the amplitudes of the received audible commands, and determine,based on the amplitude of the received audible commands, whichcontrollable device the audible command is directed towards, based on alarger amplitude.

According to a second aspect of the embodiments, a system is providedfor determining which controllable device an audible command is directedtowards, the system comprising two or more controllable devices; two ormore controlling devices, each of which is adapted to control respectiveones of the two or more controllable devices, receive the audiblecommand, digitize the received audible command, and transmit the same;and a central processor adapted to receive the transmitted digitizedaudible commands from the two or more controlling devices, and apply aspeech recognition algorithm to the received digitized audible commandsto determine which controllable device the received audible command isdirected towards.

According to the second aspect of the embodiments, the system furthercomprises the central processor being further adapted to prepare andtransmit a control command to the controllable device that the receivedaudible command was directed towards, to enact the audible command.

According to a third aspect of the embodiments, a method for determiningwhich controllable device an audible command is directed towards isprovided, the method comprising: receiving at two or more controllingdevices an audible command, the audible command directed to control atleast one of two or more controllable devices controlled by a respectiveone of the two or more controlling devices; digitizing the receivedaudible command and transmitting the same by each of the two or morecontrolling devices; receiving the transmitted digitized audiblecommands at a central processor and applying a speech recognitionalgorithm to determine which of the at least two or more controllabledevices the audible command was directed towards.

According to the third aspect of the embodiments, the method furthercomprises adding a unique controlling device identifier to the receivedaudible command at each respective controlling device; applying a timeand date stamp to the received audible command at each respectivecontrolling device; transmitting the respective time and date stampedversions of the received audible command; and receiving the transmittedtime and date stamped versions of the audible commands at a centralprocessor, the central processor adapted to receive each of thetransmitted time and date stamped versions of the audible command andperform processing based on the time and date stamp, and wherein thecentral processor is further adapted to determine the earliest time anddate stamp of each of the received audible commands by comparing each ofthe respective time and date stamps of each of the received audiblecommands; generating a command signal by the central processor andtransmitting the same to the controllable device that corresponds to thecontrolling device that had the earliest time and date stamp.

According to the third aspect of the embodiments, the method furthercomprises adding a unique controlling device identifier to the receivedaudible command at each respective controlling device; transmitting therespective audible commands with the unique controlling deviceidentifier; and receiving the transmitted audible commands at a centralprocessor, the central processor adapted to receive each of thetransmitted audible commands and applying a central processor generatedtime and date stamp to each received audible commands, and wherein thecentral processor is further adapted to determine the earliest time anddate stamp of each of the received audible commands by comparing each ofthe respective time and date stamps of each of the received audiblecommands; generating a command signal by the central processor andtransmitting the same to the controllable device that corresponds to thecontrolling device that had the earliest time and date stamp.

According to a fourth aspect of the embodiments, a method fordetermining which controllable device an audible command is directedtowards is provided, the method comprising: receiving at each of two ormore controlling devices the audible command, the audible command beingdirected to control at least one of two or more controllable devicescontrolled by a respective one of the two or more controlling devices;digitizing each of the received audible commands; attaching a time-datestamp to each of the digitized audible commands using a time-date stampgenerator that employs a time synch protocol, and further attaching aunique identifier to each of the time-date stamped digitized audiblecommands so as to uniquely correlate it to a respective controllingdevice; determining a first received digitized audible command on thebasis of an earliest time-date stamp, and further determining to whichcontrolling device the audible command is directed to on the basis ofthe unique identifier of the first received digitized audible command;performing speech recognition on the first received digitized audiblecommand to determine a command for a controllable device; and forwardingthe command to the controlling device corresponding to the firstreceived digitized audible command, the command corresponding to thespoken audible command that can be implemented on the controllabledevice controlled by the controlling device.

According to the fourth aspect of the embodiments, the method furthercomprises: receiving the command by the controlling device; andcontrolling the controllable device in accordance with the receivedcommand.

According to the fourth aspect of the embodiments, the step ofdetermining further comprises: determining a magnitude of each of thedigitized audible commands from respective controlling devices; andverifying that the magnitude of the digitized audible command with theearliest time-date stamp is equal to or larger than any other digitizedaudible commands.

According to the fourth aspect of the embodiments, the method furthercomprises: applying additional processing to determine which controllingdevice the received audible command is directed to if the magnitude ofthe received and digitized audible command with the earliest time-datestamp is less than any other received and digitized audible command.

According to the fourth aspect of the embodiments, the step of applyingadditional processing comprises: checking one or more of an occupancysensor reading, proximity sensor reading, and motion detector reading,each of which is associated with respective controlling devices todetermine which controlling device the received audible command isdirected to.

According to the fourth aspect of the embodiments, the method furthercomprises: determining that the occupancy sensor reading associated witha respective controlling device matches the received audible command inorder to determine that the received audible command should be appliedto the controllable device controlled by the respective controllingdevice.

According to the fourth aspect of the embodiments, the method furthercomprises: determining that the proximity sensor reading associated witha respective controlling device matches the received audible command inorder to determine that the received audible command should be appliedto the controllable device controlled by the respective controllingdevice.

According to the fourth aspect of the embodiments, the method furthercomprises: determining that the motion detector reading associated witha respective controlling device matches the received audible command inorder to determine that the received audible command should be appliedto the controllable device controlled by the respective controllingdevice.

According to the fourth aspect of the embodiments, the method furthercomprises: applying noise reduction processing.

According to the fourth aspect of the embodiments, the step of applyingnoise reduction to the received audible command comprises: filtering thereceived audible command in an analog domain to attenuate a firstbandwidth of noise energy.

According to the fourth aspect of the embodiments, the step of applyingnoise reduction comprises: using one or more of acoustic echocancellation filtering, direction of arrival filtering, anddirectionally adaptive beam forming filtering, to filter the digitizedaudible command in a digital domain to attenuate noise energy and toamplify audible command energy.

According to the fourth aspect of the embodiments, the noise energycomprises: noise energy generated by one or more of a fan motor, music,air conditioning noise, audio generated by multi-media presentations,and non-command words.

According to the fourth aspect of the embodiments, the method furthercomprises: receiving at least one additional audible command from atleast one additional spatially separated microphone, the at least oneadditional spatially separated microphone associated with a respectivecontrolling device; digitizing the at least one additional audiblecommand received from the at least one additional spatially separatedmicrophone; attaching a time-date stamp to the at least one digitizedadditional audible command, and further attaching a unique identifier toeach time-date stamped digitized additional audible command so as touniquely correlate it to its respective controlling device; and usingthe at least one additional digitized audible command to assist indetermining to which controlling device the audible command is directedto.

According to the fourth aspect of the embodiments, the step ofdetermining, on the basis of the time-date stamp, to which controllingdevice the digitized audible command is directed to, comprises:generating the time-date stamp using a time-date stamp generator, thetime-date stamp generator located in at least one controlling devicethat includes a plurality of microphones located at the controllingdevice; averaging all of the time-date stamps generated at thecontrolling device with a plurality of microphones; and attaching theaveraged time-date stamp to each of the plurality of digitized audiblecommands prior to the step of transmitting.

According to the fourth aspect of the embodiments, the steps ofdetermining, performing, and forwarding are performed by a centralcontroller.

According to the fourth aspect of the embodiments, the steps ofdigitizing, and attaching the time-date stamp are performed by thecontrolling device.

According to the fourth aspect of the embodiments, the controllabledevices include one or more of a sensor, lighting control device, shadedevice, audio/video device, environmental control device, securitydevice, household appliance, control device, and industrial device.

According to the fourth aspect of the embodiments, the controllingdevice comprises a keypad.

According to a fifth aspect of the embodiments, an acoustic sensornetwork is provided, comprising: two or more controllable devices; twoor more controlling devices, each of two or more controlling devicescomprising at least one respective microphone, a time-date stampgenerator, and an analog to digital converter, each of the two or morecontrolling devices being adapted to control a respective one of the twoor more controllable devices, and wherein each of the two or morecontrolling devices are further adapted to receive an audible commandthrough its respective microphone, the received audible command beingdirected to control one of the two or more controllable devicescontrolled by a respective controlling device, and wherein each of theanalog to digital converters are adapted to digitize the receivedaudible command, and wherein each of the two or more controlling devicesare further adapted to attach a time-date stamp to each of the digitizedaudible commands using the time-date stamp generator that employs a timesynch protocol and attach a unique identifier to each of the time-datestamped digitized audible commands so as to uniquely correlate thetime-date stamped digitized audible command to a respective controllingdevice; and a central controller adapted to determine, on the basis ofan earliest time-date stamp, a first received digitized audible commandand the controlling device to which the audible command is directed to,and wherein the central controller is further adapted to perform speechrecognition on the first received digitized audible command to determinea command for a controllable device, and wherein the central controlleris further adapted to forward the command to the controlling devicecorresponding to the first received digitized audible command, thecommand corresponding to the audible command that can be implemented onthe controllable device controlled by the controlling device.

According to the fifth aspect of the embodiments, the controlling devicethat receives the command is adapted to control the controllable devicein accordance with the received command.

According to the fifth aspect of the embodiments, the central controlleris further adapted to determine a magnitude of each of the digitizedaudible commands from respective controlling devices, and verify thatthe magnitude of the digitized audible command with the earliesttime-date stamp is equal to or larger than any other digitized audiblecommand signal.

According to the fifth aspect of the embodiments, the central controlleris further adapted to apply additional processing to determine whichcontrolling device the received audible command is directed to if themagnitude of the digitized audible command signal with the earliesttime-date stamp is less than any other received audible command.

According to the fifth aspect of the embodiments, the central controlleris further adapted to check one or more of an occupancy sensor reading,proximity sensor reading, and motion detector reading, each of which isassociated with respective controlling devices to determine whichcontrolling device the received audible command is directed to.

According to the fifth aspect of the embodiments, the central controlleris further adapted to determine that the occupancy sensor readingassociated with a respective controlling device matches the receivedaudible command in order to determine that the received audible commandshould be applied to the controllable device controlled by therespective controlling device.

According to the fifth aspect of the embodiments, the central controlleris further adapted to determine that the proximity sensor readingassociated with a respective controlling device matches the receivedaudible command in order to determine that the received audible commandshould be applied to the controllable device controlled by therespective controlling device.

According to the fifth aspect of the embodiments, the central controlleris further adapted to determine that the motion detector readingassociated with a respective controlling device matches the receivedaudible command in order to determine that the received audible commandshould be applied to the controllable device controlled by therespective controlling device.

According to the fifth aspect of the embodiments, the controlling devicefurther comprises a noise reduction processing circuit.

According to the fifth aspect of the embodiments, the noise reductionprocessing circuit is adapted to filter the received analog audiblecommand signal in an analog domain to attenuate a first bandwidth ofnoise energy.

According to the fifth aspect of the embodiments, the noise reductionprocessing circuit is adapted to use one or more of acoustic echocancellation filtering, direction of arrival filtering, anddirectionally adaptive beam forming filtering, to filter the digitalaudible command signal in a digital domain to attenuate noise energy andto amplify audible command energy.

According to the fifth aspect of the embodiments, the noise energycomprises: noise energy generated by one or more of a fan motor, music,air conditioning noise, audio generated by multi-media presentations,and non-command words.

According to the fifth aspect of the embodiments, the acoustic sensornetwork further comprises at least one additional spatially separatedmicrophone, adapted to receive the audible command and associated withone of the at least two controlling devices; an analog to digitalconverter associated with the at least one additional spatiallyseparated microphone, and adapted to digitize the received audiblecommand; a time-date stamp generator adapted to add a time-date stamp tothe at least one additional digitized audible command, and furtheradapted to add a unique identifier to the at least one time-datedstamped additional digitized audible command, the unique identifiercorresponding to the associated one of the controlling devices, andfurther wherein the central controller uses the at least one additionaldigitized audible command to assist in determining to which controllingdevice the audible command is directed to.

According to the fifth aspect of the embodiments, the controlling devicefurther comprises: a time-date stamp generator adapted to generate thetime-date stamp; and at least two microphones, each of which digitizesthe received audible command.

According to the fifth aspect of the embodiments, the controlling deviceis further adapted to average all of the time-date stamps generated atthe controlling device, and attach the averaged time-date stamps to eachof the plurality of digitized audible commands prior to transmitting thesame.

According to the fifth aspect of the embodiments, the controllabledevices include one or more of a sensor, lighting control device, shadedevice, audio/video device, environmental control device, securitydevice, household appliance, control device, and industrial device.

According to the fifth aspect of the embodiments, the controlling devicecomprises a keypad.

According to a sixth aspect of the embodiments, a method for determiningwhich controllable device an audible command is directed towards isprovided, the method comprising: receiving at each of two or morecontrolling devices the audible command signal, the audible commandbeing directed to control at least one of two or more controllabledevices controlled by a respective one of the two or more controllingdevices; digitizing each of the received audible command signals;attaching a unique identifier to each digitized audible command so as touniquely correlate it to a respective controlling device; determining amagnitude of each of the digitized audible command; determining adigitized audible command with the greatest magnitude, and furtherdetermining to which controlling device the audible command is directedto on the basis of the unique identifier associated with the digitizedaudible command with the greatest magnitude; performing speechrecognition on the digitized audible command with the greatestmagnitude; and forwarding a command to the controlling devicecorresponding to the digitized audible command with the greatestmagnitude, the command corresponding to the audible command that can beimplemented on the controllable device controlled by the controllingdevice.

According to the sixth aspect of the embodiments, the method furthercomprises: receiving the command by the controlling device; andcontrolling the controllable device in accordance with the receivedcommand.

According to the sixth aspect of the embodiments, the step ofdetermining further comprises: attaching a time-date stamp to thedigitized audible command; and verifying that the time-date stamp of thegreatest magnitude digitized audible command is the same or earlier thanany other digitized audible command.

According to the sixth aspect of the embodiments, the step of attachinga time-date stamp to the digitized audible command is performed by thecontrolling device that received the audible command through use of atime-date stamp generator using a time synch protocol.

According to the sixth aspect of the embodiments, the step of attachinga time-date stamp to the digitized audible command is performed by acentral controller.

According to the sixth aspect of the embodiments, the method furthercomprises: generating a test signal to determine a travel time from eachof the plurality of controlling devices to the central controller; andmodifying the time-date stamp of each received digitized audible commandsignal according to the travel time from a respective controlling deviceto the central processor, and using the modified time-date stamp toassist in determining to which controlling device the audible commandsignal is directed to.

According to the sixth aspect of the embodiments, the method furthercomprises: applying additional processing to determine which controllingdevice the audible command is directed to if the time-date stamp of thedigitized audible command with the largest magnitude is later than anyother digitized audible command.

According to the sixth aspect of the embodiments, the step of applyingadditional processing comprises: checking one or more of an occupancysensor reading, proximity sensor reading, and motion detector reading,each of which is associated with respective controlling devices todetermine which controlling device the audible command is directed to.

According to the sixth aspect of the embodiments, the method furthercomprises: determining that the occupancy sensor reading associated witha respective controlling device matches the audible command in order todetermine that the audible command should be applied to the controllabledevice controlled by the respective controlling device.

According to the sixth aspect of the embodiments, the method furthercomprises: determining that the proximity sensor reading associated witha respective controlling device matches the audible command in order todetermine that the audible command should be applied to the controllabledevice controlled by the respective controlling device.

According to the sixth aspect of the embodiments the method furthercomprises: determining that the motion detector reading associated witha respective controlling device matches the audible command in order todetermine that the audible command should be applied to the controllabledevice controlled by the respective controlling device.

According to the sixth aspect of the embodiments, the step of attachinga time-date stamp comprises: generating the time-date stamp using atime-date stamp generator, the time-date stamp generator located in atleast one controlling device that includes a plurality of microphoneslocated at the controlling device; averaging all of the time-date stampsgenerated at the controlling device with a plurality of microphones; andattaching the averaged time-date stamp to each of the plurality ofdigitized audible commands prior to the step of transmitting.

According to the sixth aspect of the embodiments, the method furthercomprises: applying noise reduction processing, and wherein the step ofapplying noise reduction to the audible command comprises: filtering thereceived audible command signal in an analog domain to attenuate a firstbandwidth of noise energy.

According to the sixth aspect of the embodiments, the step of applyingnoise reduction comprises: using one or more of acoustic echocancellation filtering, direction of arrival filtering, anddirectionally adaptive beam forming filtering, to filter the digitizedaudible command in a digital domain to attenuate noise energy and toamplify audible command energy.

According to the sixth aspect of the embodiments the noise energycomprises: noise energy generated by one or more of a fan motor, music,air conditioning noise, audio generated by multi-media presentations,and non-command words.

According to the sixth aspect of the embodiments, the method furthercomprises: receiving at least one additional audible command from atleast one additional spatially separated microphone, the at least oneadditional spatially separated microphone associated with a respectivecontrolling device; digitizing the at least one additional audiblecommand received from the at least one additional spatially separatedmicrophone; attaching a time-date stamp to the at least one additionaldigitized audible command, and further attaching a unique identifier toeach time-date stamped additional digitized audible command so as touniquely correlate it to its respective controlling device; and usingthe at least one additional digitized audible command in the step ofdetermining to which controlling device the audible command is directedto.

According to the sixth aspect of the embodiments, the steps ofdetermining a magnitude, determining to which controlling device theaudible command is directed to, performing, and forwarding are performedby a central controller.

According to the sixth aspect of the embodiments, the steps ofreceiving, digitizing, attaching the time-date stamp, and determining amagnitude, are performed by the controlling device.

According to the sixth aspect of the embodiments the controllabledevices include one or more of a sensor, lighting control device, shadedevice, audio/video device, environmental control device, securitydevice, household appliance, control device, and industrial device.

According to the sixth aspect of the embodiments, the controlling devicecomprises a keypad.

According to a seventh aspect of the embodiments, an acoustic sensornetwork is provided, comprising: two or more controllable devices; twoor more controlling devices, each of two or more controlling devicescomprising at least one respective microphone, and an analog to digitalconverter, each of the two or more controlling devices being adapted tocontrol a respective one of the two or more controllable devices, andwherein each of the two or more controlling devices are further adaptedto receive an audible command through at least one respectivemicrophone, the received audible command being directed to control oneof the two or more controllable devices controlled by a respectivecontrolling device, and wherein each of the analog to digital convertersare adapted to digitize the received audible command, and wherein eachof the two or more controlling devices are further adapted to attach aunique identifier to each of the digitized audible commands so as touniquely correlate it to a respective controlling device and transmitthe same; and a central controller adapted to receive each of thetransmitted digitized audible commands, determine a magnitude of each ofthe digitized audible commands, determine a digitized audible commandwith the greatest magnitude, and further determine to which controllingdevice the audible command is directed to on the basis of the uniqueidentifier associated with the digitized audible command with thegreatest magnitude, and wherein the central controller is furtheradapted to perform speech recognition on the digitized audible commandwith the greatest magnitude to determine a command for a controllabledevice, and wherein the central controller is further adapted to forwardthe command to the controlling device corresponding to the digitizedaudible command with the greatest magnitude, the command correspondingto the audible command that can be implemented on the controllabledevice controlled by the controlling device.

According to the seventh aspect of the embodiments, the controllingdevice that receives the command is adapted to control the controllabledevice in accordance with the received command.

According to the seventh aspect of the embodiments, each of thecontrolling devices are further adapted to attach a time-date stamp tothe digitized audible commands through use of a time-date stampgenerator using a time synch protocol, and wherein the centralcontroller is further adapted to verify that the time-date stamp of thedigitized audible command with the greatest magnitude is the same orearlier than the time-date stamp of any other digitized received audiblecommand signal.

According to the seventh aspect of the embodiments, the centralcontroller is further adapted to generate a test signal to determine atravel time from each of the plurality of controlling devices to thecentral controller, modify the time-date stamp of each receiveddigitized audible command signal according to the travel time from arespective controlling device to the central processor, and use themodified time-date stamp to assist in determining to which controllingdevice the audible command signal is directed to.

According to the seventh aspect of the embodiments, the centralcontroller is further adapted to apply additional processing todetermine which controlling device the received audible command isdirected to if the time-date stamp of the digitized audible commandsignal with the greatest magnitude is later than the time-date stamp anyother digitized audible command.

According to the seventh aspect of the embodiments, the centralcontroller is further adapted to check one or more of an occupancysensor reading, proximity sensor reading, and motion detector reading,each of which is associated with respective controlling devices todetermine which controlling device the received audible command isdirected to.

According to the seventh aspect of the embodiments, the centralcontroller is further adapted to determine that the occupancy sensorreading associated with a respective controlling device matches thereceived audible command in order to determine that the received audiblecommand should be applied to the controllable device controlled by therespective controlling device.

According to the seventh aspect of the embodiments, the centralcontroller is further adapted to determine that the proximity sensorreading associated with a respective controlling device matches thereceived audible command in order to determine that the received audiblecommand should be applied to the controllable device controlled by therespective controlling device.

According to the seventh aspect of the embodiments, the centralcontroller is further adapted to determine that the motion detectorreading associated with a respective controlling device matches thereceived audible command in order to determine that the received audiblecommand should be applied to the controllable device controlled by therespective controlling device.

According to the seventh aspect of the embodiments, the controllingdevice further comprises a noise reduction processing circuit.

According to the seventh aspect of the embodiments, the noise reductionprocessing circuit is adapted to filter the received audible command inan analog domain to attenuate a first bandwidth of noise energy.

According to the seventh aspect of the embodiments, the noise reductionprocessing circuit is adapted to use one or more of acoustic echocancellation filtering, direction of arrival filtering, anddirectionally adaptive beam forming filtering, to filter the digitizedaudible command in a digital domain to attenuate noise energy and toamplify audible command energy.

According to the seventh aspect of the embodiments, the noise energycomprises noise energy generated by one or more of a fan motor, music,air conditioning noise, audio generated by multi-media presentations,and non-command words.

According to the seventh aspect of the embodiments, the acoustic sensornetwork further comprises: at least one additional spatially separatedmicrophone, adapted to receive the audible command, and associated withone of the at least two controlling devices; an analog to digitalconverter associated with the at least one spatially separatedmicrophone, and adapted to digitize the received audible command; atime-date stamp generator adapted to add a time-date stamp to theadditional digitized audible command, and further adapted to add aunique identifier to the additional digitized audible command, theunique identifier corresponding to the associated one of the controllingdevices, and further wherein the central controller uses the at leastone additional digitized audible command to assist in determining towhich controlling device the audible command is directed to.

According to the seventh aspect of the embodiments, the controllingdevice further comprises: a time-date stamp generator adapted togenerate the time-date stamp; and at least two microphones, each ofwhich digitizes the received audible command.

According to the seventh aspect of the embodiments, the controllingdevice is further adapted to average all of the time-date stampsgenerated at the controlling device, and attach the averaged time-datestamps to each of the plurality of digitized audible commands prior totransmitting the same.

According to the seventh aspect of the embodiments, the controllabledevices include one or more of a sensor, lighting control device, shadedevice, audio/video device, environmental control device, securitydevice, household appliance, control device, and industrial device.

According to the seventh aspect of the embodiments, the controllingdevice comprises a keypad.

According to an eighth aspect of the embodiments, a microphone system isprovided, comprising: one or more microphones each of which is adaptedto receive acoustic audio signals and convert the same from acousticalenergy signals into electrical audio signals and output the same asmicrophone output audio signals; and an acoustic echo cancellation (AEC)device adapted to receive the microphone output audio signals, andwherein the AEC device also receives as a second input a reference audiosignal from a network, and wherein the AEC device is further adapted tocancel substantially all of the reference audio signal from themicrophone output audio signals and output the same as a corrected audiosignal, and wherein the reference audio signal comprises an audio signalgenerated by an external audio system.

According to the eighth aspect of the embodiments, the AEC device isfurther adapted to subtract the reference input signal from themicrophone output audio signals when cancelling, and outputting theresult as the corrected audio signal.

According to the eighth aspect of the embodiments, the microphone systemfurther comprises a direction detection and beamforming (DDB) circuitadapted to receive two or more corrected audio signals and combine thesame into a single audio output signal.

According to the eighth aspect of the embodiments, the DDB is furtheradapted to encrypt the corrected audio signal prior to being output fromthe DDB.

According to the eighth aspect of the embodiments, the reference audiosignal comprises: an audio signal that is provided to externalamplifiers and speakers.

According to the eighth aspect of the embodiments, the externalamplifiers and speakers are located within an audible detection radiusof the AEC circuit.

According to the eighth aspect of the embodiments, the externalamplifiers and speakers are located within hearing distance of the twoor more microphones.

According to the eighth aspect of the embodiments, the microphone systemfurther comprises a keyword recognition device adapted to detect thepresence of a spoken keyword in the corrected audio signal.

According to the eighth aspect of the embodiments, the microphone systemis further adapted to initiate transmission of the corrected audiosignal to a voice recognition server through a network followingdetection of the spoken keyword, and to terminate transmission of thecorrected audio signal upon the occurrence of a termination event.

According to the eighth aspect of the embodiments, the termination eventcomprises: at least one of a timeout condition, and one or moretermination words.

According to the eighth aspect of the embodiments, the received acousticaudio signals comprises a combination of a desired spoken audio signaland a delayed version of undesired audio signals, and wherein thereference audio signal comprises an undelayed version of the undesiredaudio signals, and further wherein the corrected audio signal comprisessubstantially only the desired spoken audio signal.

According to the eighth aspect of the embodiments, the microphone systemfurther comprises: a delay circuit adapted to delay substantially all ofthe undesired audio signals prior to the undesired audio signals beingbroadcast by one or more speakers within hearing distance of themicrophone system.

According to the eighth aspect of the embodiments, the microphone systemfurther comprises: a voice recognition system adapted to receive andperform a speech recognition analysis on the corrected audio signal.

According to the eighth aspect of the embodiments, the voice recognitionsystem is further adapted to respond to the recognized corrected audiosignal.

According to the eighth aspect of the embodiments, the microphone systemis adapted to be used in a full duplex intercom mode.

According to the eighth aspect of the embodiments, the microphone systemis adapted to enable full duplex intercom conversations between a firstmicrophone system and one or more additional microphones.

According to the eighth aspect of the embodiments, the microphone systemis adapted to be used as a telephone system.

According to the eighth aspect of the embodiments, the microphone systemis adapted to enable telephone conversations between a first microphoneand one or more additional microphones.

According to the eighth aspect of the embodiments, the microphonecomprises an Ethernet network device.

According to the eighth aspect of the embodiments, the Ethernet networkdevice is adapted to receive power over a power-over-Ethernet interface.

According to the eighth aspect of the embodiments, the network isassociated with a voice recognition system.

According to the eighth aspect of the embodiments, the reference audiosignal is encrypted prior to being transmitted from the network to theAEC device.

According to a ninth aspect of the embodiments, a microphone system isprovided, comprising: one or more microphones each of which is adaptedto receive acoustic audio signals and convert the same from acousticalenergy signals into electrical audio signals and output the same asmicrophone output audio signals; and an acoustic echo cancellation (AEC)device adapted to receive the microphone output audio signals, andwherein the AEC device also receives as a second input a reference audiosignal from a network, and wherein the AEC device is further adapted tosubtract the reference input signal from the microphone output audiosignals, and output the same as a corrected audio signal, and whereinthe reference audio signal comprises an audio signal generated by anexternal audio system.

According to the ninth aspect of the embodiments, the microphone systemfurther comprises: a direction detection and beamforming (DDB) circuitadapted to receive two or more corrected audio signals and combine thesame into a single audio output signal.

According to the ninth aspect of the embodiments, the DDB is furtheradapted to encrypt the corrected audio signal prior to being output fromthe DDB.

According to the ninth aspect of the embodiments, the received acousticaudio signals comprises a combination of a desired spoken audio signaland a delayed version of undesired audio signals, and wherein thereference audio signal comprises an undelayed version of the undesiredaudio signals, and further wherein the corrected audio signal comprisessubstantially only the desired spoken audio signal.

According to a tenth aspect of the embodiment, a microphone system isprovided, comprising: one or more microphones each of which is adaptedto receive acoustic audio signals and convert the same from acousticalenergy signals into electrical audio signals and output the same asmicrophone output audio signals; an acoustic echo cancellation (AEC)device adapted to receive the microphone output audio signals, andwherein the AEC device also receives as a second input a reference audiosignal from a network, and wherein the AEC device is further adapted tosubtract the reference input signal from the microphone output audiosignals, and output the same as a corrected audio signal, and whereinthe reference audio signal comprises an audio signal generated by anexternal audio system, and further wherein the received acoustic audiosignals comprises a combination of a desired spoken audio signal and adelayed version of undesired audio signals, and wherein the referenceaudio signal comprises an undelayed version of the undesired audiosignals, and further wherein the corrected audio signal comprisessubstantially only the desired spoken audio signal; and a delay circuitadapted to delay substantially all of the undesired audio signals priorto the undesired audio signals being broadcast by one or more speakerswithin hearing distance of the microphone system.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and features of the embodiments will becomeapparent and more readily appreciated from the following description ofthe embodiments with reference to the following figures. Differentaspects of the embodiments are illustrated in reference figures of thedrawings. It is intended that the embodiments and figures disclosedherein are to be considered to be illustrative rather than limiting. Thecomponents in the drawings are not necessarily drawn to scale, emphasisinstead being placed upon clearly illustrating the principles of theaspects of the embodiments. In the drawings, like reference numeralsdesignate corresponding parts throughout the several views.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 illustrates a block diagram of a control network for controllingone or more controllable devices in a home, business, or enterpriseenvironment according to aspects of the embodiments.

FIG. 2 illustrates a block diagram of a controller for use in thecontrol network of FIG. 1 according to aspects of the embodiments.

FIG. 3 illustrates a block diagram of a gateway for use in the controlnetwork of FIG. 1 according to aspects of the embodiments.

FIG. 4 illustrates a block diagram of a personal electronic device foruse with the control system of FIG. 1 according to aspects of theembodiments.

FIG. 5 illustrates a block diagram of a wall mount keypad for use in thecontrol network as shown in FIG. 1, wherein the wall mount keypad can beused as part of an acoustic sensory network according to aspects of theembodiments.

FIG. 6 illustrates a simplified view of the generation and detection ofsound waves as used in the aspects of the embodiments.

FIG. 7 illustrates a plan view of a floor of a house in which the systemand method for determining which controllable device an audible commandis directed to can be used according to aspects of the embodiments.

FIG. 8 illustrates a flow diagram of a method for determining whichcontrollable device an audible command is directed to according toaspects of the embodiments.

FIG. 9 illustrates processing and memory components/circuitry of one ormore of the personal electronic device 104 of FIG. 4, gateway device 114of FIG. 3, controller 116 of FIG. 2, and any other devices that uses oneor more processors as described herein that uses software and/orapplications to perform various functions and actions as describedherein according to aspects of the embodiments.

FIG. 10 illustrates several audio processing blocks that can occurwithin either or both of an audio processing board and a voicerecognition system-on-a-chip circuit according to aspects of theembodiments.

FIG. 11 illustrates a conventional voice recognition system, a personspeaking, and an unrelated multi-media audio system broadcasting unknownmulti-media audio, as well as unrelated multi-media audio broadcast by avideo distribution system.

FIG. 12 illustrates a block diagram view of an external audiocompensated voice recognition system (EAC-VRS) according to aspects ofthe embodiments, wherein previously unknown multi-media audio, broadcastby an unrelated multi-media audio system, and unrelated multi-mediaaudio broadcast by a video distribution system, becomes knownmulti-media audio, and the EAC-VRS, according to aspects of theembodiments, can take into account known multi-media audio to enhancethe recognition of voice audio generated by person speaking, accordingto aspects of the embodiments.

FIG. 13 illustrates a block diagram of a remotely located networkmicrophone device using micro-electromechanical technology for use inthe EAC-VRS of FIG. 12 according to aspects of the embodiments.

FIG. 14 illustrates a block diagram of a further embodiment of thenetwork microphone device of FIG. 13 using micro-electromechanicaltechnology according to an aspect of the embodiments.

FIG. 15 illustrates a block diagram of an HDMI audio extractor devicefor use in the EAC-VRS of FIG. 12 according to aspects of theembodiments.

FIG. 16 illustrates a timing diagram of delayed known multi-media audio,un-delayed multi-media audio, and unknown audio from a speaker and/orother sources according to aspects of the embodiments.

FIG. 17 illustrates a simplified block diagram illustrating theprinciple of determining an appropriate time delay between thebroadcasting of known audio from one or more speakers and receipt by amicrophone associated with an acoustic echo cancellation device whereinknowledge of the delay increases the efficacy of echo cancellation andother audio system processes according to aspects of the embodiments.

FIG. 18 illustrates a block diagram view of an external audiocompensated voice recognition system (EAC-VRS) according to furtheraspects of the embodiment.

FIG. 19 illustrates a detailed block diagram of the EAC-VRS NW processorreferenced in regard to FIG. 12 according to aspects of the embodiments.

FIG. 20 illustrates a detailed block diagram of a down-mixer device anddelay according to aspects of the embodiments.

FIG. 21 illustrates a flow chart of a method for compensating for knownaudio in regard to unknown audio, so that the unknown audio can more bemore clearly determined using an external audio compensated voicerecognition system (EAC-VRS) according to aspects of the embodiments.

FIG. 22 illustrates an audio extraction and delay device according toaspects of the embodiments.

DETAILED DESCRIPTION

The embodiments are described more fully hereinafter with reference tothe accompanying drawings, in which embodiments of the inventive conceptare shown. In the drawings, the size and relative sizes of layers andregions may be exaggerated for clarity. Like numbers refer to likeelements throughout. The embodiments may, however, be embodied in manydifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the inventive concept to those skilled in the art.The scope of the embodiments is therefore defined by the appendedclaims. The detailed description that follows is written from the pointof view of a control systems company, so it is to be understood thatgenerally the concepts discussed herein are applicable to varioussubsystems and not limited to only a particular controlled device orclass of devices, such as home controllable devices.

Reference throughout the specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with an embodiment is included inat least one embodiment of the embodiments. Thus, the appearance of thephrases “in one embodiment” on “in an embodiment” in various placesthroughout the specification is not necessarily referring to the sameembodiment. Further, the particular feature, structures, orcharacteristics may be combined in any suitable manner in one or moreembodiments.

LIST OF REFERENCE NUMBERS FOR THE ELEMENTS IN THE DRAWINGS IN NUMERICALORDER

The following is a list of the major elements in the drawings innumerical order.

-   100 Control Network-   102 Institute of Electrical and Electronic Engineers standard (IEEE)    802.15.4 Low Rate Wireless Personal Area Network (LR-PAN)    (Communication Network)-   104 Portable Electronic Device (PED)-   106 Control Point/User Interface/Keypad (Keypad)-   108 Sensor-   110 Lighting Control Device (Lighting Device)-   112 Shade Control Device (Shade Device)-   114 Gateway Device-   115 First Antenna-   116 Controller (Controller)-   117 Second Antenna-   118 Audio/Video (AV) Device-   119 Third Antenna-   120 Heating Ventilation and Air Conditioning (HVAC) Device-   122 Security Device-   124 Household Appliances-   126 Control Device-   128 Industrial Device-   130 Repeaters-   132 Internet-   134 Local Area Network (LAN)-   136 Router/Firewall-   202 Central Processor Unit (CPU)-   204 Nonvolatile Storage-   206 Main Memory-   208 Network Interfaces-   210 Wired I/O Interface-   212 Low Rate Wireless Personal Area Network (LR-WPAN) Transceiver    (LR-WPAN Transceiver) (IEEE 802.15.4)-   214 Wireless Local Area Network (WLAN) Transceiver (WLAN    Transceiver) (IEEE 802.11)-   216 3G/4G/LTE Wireless Wide Area Network (WWAN) Cellular Transceiver    (Cellular Transceiver)-   218 Programmable Relay Ports-   220 Internal Bus-   222 Audible Command Processing and Determination Program-   302 Network Interface-   304 Power On/Off LED-   306 Network Activity Indicator LED-   308 Activity Indicator LED-   310 Acquire Button-   312 Setup Button-   314 Wireless Transceiver-   316 Processor-   318 Internal Bus-   402 Central Processing Unit-   406 Location Sensing Circuitry-   408 User Interface-   410 Display-   412 Non-volatile Storage-   414 Main Memory-   416 NFC Interface-   418 Accelerometers-   420 Camera-   502 Display/Touch Panel-   504 Microphone-   505 Audio Processing Board-   506 Pre-amplifier-   508 Analog-to-Digital Converter (ADC)-   510 60 Hz Notch Filter-   512 Processor-   516 Internal Bus-   518 LAN/Ethernet Connector (IEEE 802.3)-   520 Voice Recognition (VR) System-on-a-Chip (SoC) Circuit (VR SoC    Circuit)-   702 Proximity Sensor-   800 Method for Determining Which Controllable Device an Audible    Command is Directed Towards-   802-812 Method Steps of Method 800-   900 Personal Computer/Laptop/Tablet/Personal Electronic Device    (PED)/Server (PC)-   902 Integrated Display/Touch-Screen (laptop/tablet etc.)-   904 Internal Data/Command Bus (Bus)-   906 Processor Internal Memory-   908 Processor(s)-   910 Universal Serial Bus (USB) Port-   911 Ethernet Port-   912 Compact Disk (CD)/Digital Video Disk (DVD) Read/Write (RW)    (CD/DVD/RW) Drive-   914 Floppy Diskette Drive-   916 Hard Disk Drive (HDD)-   918 Read-Only Memory (ROM)-   920 Random Access Memory (RAM)-   922 Video Graphics Array (VGA) Port or High Definition Multimedia    Interface (HDMI)-   924 External Memory Storage Device-   932 Processor Board/PC Internal Memory (Internal Memory)-   934 Flash Drive Memory-   936 CD/DVD Diskettes-   938 Floppy Diskettes-   940 Executable Software Programming Code/Application (Application,    or “App”)-   956 Universal Serial Bus (USB) Cable-   1002 Analog Processing Circuit-   1004 Time-Date Stamp Generator-   1006 Acoustic Echo Cancellation Block-   1008 Direction of Arrival Block-   1010 Directionally Adaptive Beam Forming Block-   1100 Conventional Voice Recognition System (CVRS)-   1102 Conventional Voice Recognition Circuitry (CVRC)-   1104 Voice Recognition System (VRS) Processor-   1106 Network Connection-   1108 Speaker Cable-   1110 Voice Recognition Server-   1114 Speaker-   1116 Microphone (Mic)-   1118 Multi-Media Audio-Known (MMA_(K))-   1120 Multi-Media Audio-Unknown (MMA_(U))-   1122 Person(s)-   1124 Unknown Voice Audio-   1126 Internet (Network, LAN, WAN, PAN, etc.)-   1128 HDMI Transceiver-   1130 HDMI Cable-   1132 Audio Sound Bar-   1134 Video Display-   1138 External Multi-Media Audio System (MMAS)-   1150 Video Distribution System (VDS)-   1200 External Audio Compensated (EAC) Voice Recognition System (EAC    VRS)-   1202 External Audio Compensated Voice Recognition Circuitry    (EAC-VRSC)-   1204 External Audio Compensated Voice Recognition System Network    Audio Processor (EAC-VRS NW Processor)-   1206 HDMI Audio Extractor Device (with Programmable Delay)-   1208 Extracted Audio Transceiver-   1210 Audio Amplifier-   1212 Remotely Located Microphone (Network Microphone Device (NMD))-   1214 Remotely Located Microphone Interface-   1218 Transmitted Audio Signal-   1220 Video Distribution System-   1222 Audio Extraction and Delay Device-   1302 Direction Detection and Beamforming (DDB) Circuit-   1304 Acoustic Echo Cancellation (AEC) Circuit-   1306 Micro-Electrical-Mechanical Systems (MEMs) Microphone-   1308 Network Interface Device (NID)-   1502 HDMI Transceiver-   1504 HDMI Video and/or Audio Delay (Programmable)-   1506 HDMI Audio Extractor-   1800 External Audio Compensated Voice Recognition System with    Surround Sound Stereo System (EAC-VRS-S)-   1802 Surround Sound Stereo Source-   1804 Surround Sound Audio Transfer Cable (Audio Cable)-   1806 Speaker-   1808 Surround Sound Audio Transfer Audio Interface (Audio Interface)-   1810 Surround Sound Audio Down-Mixer Device (Audio Down-Mixer    Device)-   2002 Surround Sound Stereo Down-Mixer Circuit (Down-Mixer Circuit)-   2100 Method for Compensating for Known Audio in regard to Unknown    Audio Using an External Audio Compensated Voice Recognition System-   2102-2114 Method Steps of Method 2100-   2202 Audio Transceiver/Digitizer-   2204 Audio Delay

List of Acronyms Used in the Specification in Alphabetical Order

The following is a list of the acronyms used in the specification inalphabetical order.

-   3G Third Generation Cellular Telecommunications Network-   4G Fourth Generation Cellular Telecommunications Network-   ACPD Audio Command Processing and Determination-   ADC Analog-to-Digital Converter-   AEC Acoustic Echo Cancellation-   App Application-   ASIC Application Specific Integrated Circuit-   ASN Acoustic Sensory Network-   A/V Audio Video-   AVB Audio Video Bridging-   BIOS Basic Input-Output System-   CD Compact Disk-   CISC Complex Instruction Set-   cm Centimeter-   CPU Central Processing Unit-   CRT Cathode Ray Tubes-   CVRS Conventional Voice Recognition System-   DABF Directionally Adaptive Beam Forming-   DAS Digital Audible Signal-   DDB Direction Detection and Beamforming-   DHCP Dynamic Host Communication Protocol-   DNS Domain Name Service-   DOA Direction of Arrival-   DVD Digital Versatile Disk-   EAC-VRS External Audio Compensated Voice Recognition System-   EAC-VRSC External Audio Compensated Voice Recognition System    Circuitry-   EAC-VRS-S External Audio Compensated Voice Recognition System with    Surround Sound Stereo System-   EDGE GSM Evolution-   EEPROM Electrically Erasable Programmable Read Only Memory-   EGPRS Enhanced GPRS-   ERL Echo Return Loss-   FPGA Field Programmable Gate Array-   FPS Feet Per Second-   GPRS general packet radio service-   GPS Global Positioning System-   GSM Global System for Mobile Communications-   GUI Graphical User Interface-   HDD Hard Disk Drive-   HDMI High Definition Multi-Media-   http Hyper-Text Transport Protocol-   HVAC Heating Ventilation and Air Conditioning-   Hz Hertz-   IC Integrated Circuit-   I/O Input/Output-   IEEE Institute of Electrical and Electronic Engineers-   IMT International Mobile Telecommunications-   IP Internet Protocol-   IR Infrared-   IrDA Infra-Red Data Association-   ISO International Standards Organization-   kb/s Kilo-byte per second-   LAN Local Area Network-   LCD Liquid Crystal Display-   LED Light Emitting Diode-   LMS Least Mean Square(s)-   LR-PAN Low Rate Personal Area Network-   LR-WPAN Low Rate Wireless Personal Area Network-   LTE Long Term Evolution-   MEMS Microelectromechanical System-   Mic Microphone-   mb/s Mega-byte per second-   mm Milli-meter-   MMA Multi-Media Audio-   MMAS Multi-Media Audio System-   MMA_(K) Multi-Media Audio—Known-   MMA_(U) Multi-Media Audio—Unknown-   NFC Near Field Communications-   NIC Network Interface Card-   NICR Network Interface Controller-   NID Network Interface Device-   NMD Network Microphone Device-   NWC Network Controller-   NWI Network Interface-   OCR Optical Character Recognition-   OLED Organic LED-   OS Operating System-   OSI Open Source Initiative-   PAN Personal Area Network-   PC Personal Computer-   PCM Pulse Code Modulation-   PDA Personal Digital Assistant-   PDM Pulse Density Modulation-   PED Personal Electronic Device-   PoE Power over Ethernet-   PSTN Public Switched Telephone Network-   RAM Random Access Memory-   RCD Remote Control Devices-   RF Radio Frequency-   RFID Radio Frequency Identification-   RISC Reduced Instruction Set Processor-   ROM Read Only Memory-   RTP Real Time Protocol-   RW Read-Write-   SC Single Carrier-   SIP Session Initiation Protocol-   SoC System-on-a-Chip-   SRA Speech Recognition Algorithm-   SNR Signal-to-Noise Ratio-   USB Universal Serial Bus-   UVPROM Ultra-Violet Light Programmable Read Only Memory-   VDC Voltage, Direct Current-   VDS Video Distribution System-   VoIP Voice over Internet Protocol-   VR Voice Recognition-   VRS Voice Recognition System-   VRSv Voice Recognition Server-   WAN Wide Area Network-   WLAN Wireless Local Area Network-   Wi-Fi IEEE 802.11n Wireless Communication Standard (Where “n”    includes, “a,” “b,” or “g,” among others)-   μs/ft Micro-seconds per foot

The different aspects of the embodiments described herein pertain to thecontext of a home, office, or enterprise location control network, butis not limited thereto, except as may be set forth expressly in theappended claims.

For 40 years Creston Electronics Inc., of Rockleigh, N.J., has been theworld's leading manufacturer of advanced control and automation systems,innovating technology to simplify and enhance modern lifestyles andbusinesses. Crestron designs, manufactures, and offers for saleintegrated solutions to control audio, video, computer, andenvironmental systems. In addition, the devices and systems offered byCrestron streamlines technology, improving the quality of life incommercial buildings, universities, hotels, hospitals, and homes, amongother locations. Accordingly, the systems, methods, and modes of theaspects of the embodiments described herein, as embodied as controlnetwork 100, and its constituent components, can be manufactured byCrestron Electronics, Inc., located in Rockleigh, N.J.

FIG. 1 illustrates a block diagram of control network 100 that includescontrollable devices, monitoring devices, and active devices accordingto aspects of the embodiments. Control network 100 comprises portableelectronic device (PED) 104, control point (e.g., keypad) 106, gatewaydevice (gateway) 114, controller (controller) 116, and one or morecontrollable devices such as, but not limited to, sensors 108, lightingcontrol devices (lighting device) 110, shade control devices (shadedevice) 112, audio/video (A/V) devices 118, heating ventilation and airconditioning (HVAC) devices 120, and security devices 122. As those ofskill in the art can appreciate, there can be one or more of each of thecontrollable devices, and controller 116, PED 104, keypad 106, andgateway 114. According to further aspects of the embodiments, gateway114 and controller 116 can be part of the same device, as the dashedline box around the two indicates. According to further aspects of theembodiments, while ostensible all or substantially all of thecontrollable devices will be wireless devices, one or more can beconnected to gateway 114 and/or controller 116 by cabling (not shown).

Also shown in FIG. 1 are first antenna 115, second antenna 117, andthird antenna 119. First antenna 115 is designed to work in thefrequency band appropriate for Institute of Electrical and ElectronicEngineers (IEEE) standard 802.11n, where n can be one of “a,” “b,” and“g,” among other versions of the standard (herein after referred to as“802.11”). As those of skill in the art can appreciate, the IEEE 802.11standards encompass wireless local area networks (LANs), in this case,those that are referred to as “Wi-Fi” networks. Thus, first antenna 115is an antenna capable of transceiving Wi-Fi signals. First antenna 115is therefore included as part of PED 104, controller 116, whichcommunicate via communication network 134, described in greater detailbelow, as well as any of the devices 106, 108, 110, 112, 118, 120, 122,124, 126, 128, and 130. Each component that includes first antenna 115also can include a suitably arranged transceiver, such as a Wi-Fitransceiver, which can process signals for Wi-Fi (IEEE 802.11)transmission and reception thereof as well.

Second antenna 117 is designed to work in the frequency band appropriatefor IEEE standard 802.15.n, where n can be one of 3, 4, 5, 6, amongother versions of the standard (herein after referred to as “802.15”).As those of skill in the art can appreciate, the IEEE 802.15 standardsencompass low rate wireless personal area wireless networks (LR WPANs).In this case, the LR WPAN can be one those that are referred to as“ZigBee” networks, or, according to further aspects of the embodiments,an Infinet® as designed and manufactured by Crestron Electronics, Inc.,of Rockleigh, N.J. (among other types of LR WPANs, which can includeWirelessHart, Mi-Wi, and Thread, among others). Thus, second antenna 117is an antenna capable of transceiving ZigBee or Infinet signals.Included in any of the devices that includes second antenna 117 (whichcan include one or more of devices 104, 106, 108, 110, 112, 114, 116,118, 120, 122, 124, 126, 128, and 130) is a suitably arrangedtransceiver, such as IEEE 802.15.4 LR-WPAN transceiver (LR-WPANtransceiver) 214 that can process signals for ZigBee/Infinettransmission and reception thereof as well. All of the othercontrollable devices can also utilize such wireless communicationsdevices, so one, some, or all of them can also include second antenna117, and be also equipped with a suitable transceiver, for substantiallysimilar purposes as that of gateway device 114, among others, such askeypad 106.

According to further aspects of the embodiments, each of the devices ofnetwork 100, which can include devices 104, 106, 108, 110, 112, 114,116, 118, 120, 122, 124, 126, 128, and 130, can further include thirdantenna 119, which is adapted to work with one or more of thirdgeneration (3G), fourth generation (4G), and long term evolution (LTE)cellular (cellular) transceiver 216. Thus, controller 116 includes thirdantenna 119 and cellular transceiver 216 according to aspects of theembodiments.

As those of skill in the art can appreciate, and in fulfillment of thedual purposes of clarity and brevity, a more detailed discussion of theinternal operation of controller 116 is not needed to understand thevarious aspects of the embodiments described herein, and therefore hasbeen omitted from this discussion herein. However, such detaileddiscussion can be found in the co-pending related U.S. Non-provisionalPatent Application referenced above, the entire contents of which arehereby incorporated herein in its entirety.

Control network 100 further comprises IEEE 802.15.4 communicationnetwork (communication network) 102. As those of skill in the art canappreciate, there can be one or more of each of the controllable orcontrolling devices described above in network 100, and even two or morecommunication networks 102 a,b according to aspects of the embodiments.Control network 100 can further include LAN 134 (which can be an IEEE802.3 communication network (e.g., Ethernet), router/firewall 136, andinternet 132. As shown in FIG. 1, PED 104 can access control network 100through internet 132 and/or LAN 134. In the former, a router/firewall136 can be used to protect control network 100 and direct commands fromPED 104 to the remaining components of control network 100, as well asprovide feedback information to PED 104 from the devices of controlnetwork 100. As those of skill in the art can appreciate, a firewall asystem designed to prevent unauthorized access to or from a privatenetwork. Firewalls can be implemented in either hardware or software, ora combination of both. A router is a device that forwards data packetsalong networks. A router can be connected to at least two networks andare located at gateways.

According to further aspects of the embodiments, sensors 108 provideinformation to the various hardware and software components of thesystem and method of the aspects of the embodiments that can be used toascertain the location, movements, and mannerisms of the users ofcontrol network 100. That is, sensors 108 can be used in helping todetermine patterns of usage, and also to augment decision makingcapabilities in determining what actions to take (e.g., open or closeshades based on occupancy (or lack thereof)), or what actions not totake, as the case may be, according to aspects of the embodiments.

According to aspects of the embodiments, the one or more controllabledevices comprise lighting control device (lighting device) 110, whichcan include devices such as a lighting dimmer, and shade control device(shade device) 112, which can include devices such as a shade motor. Itshould be understood that the controllable devices are not limited to adimmer and a shade motor. For example, lighting device 110 can be aswitch or a relay panel, and shade device 112 can be a drapery motor ora smart window film. Additionally, those of skill in the art canappreciate that the controllable devices are not limited to lightingcontrol devices and shade control devices. For example, the controllabledevices can be: A/V devices 118 that can include one or more of contentsources (audio source, video source), content sinks (stereos withspeakers, televisions, and the like), video recorders, audio receivers,speakers, projectors, and the like; Lighting devices 110 that caninclude one or more of lamps, ballasts, light emitting diode (LED)drivers; HVAC devices 120 that can include one or more of thermostats,occupancy sensors, air conditioning units, heating units, filtrationsystems, fans, humidifiers, and the like; Shading devices 112 that caninclude one or more of motorized window treatments, dimmable windows,and the like; Security devices 122 that can include one or more ofsecurity cameras, monitors, door locks, and the like; Householdappliances 124 that can include one or more of refrigerators, ovens,blenders, microwaves, and the like; Control devices 126 that can includeone or more of switches, relays, current limiting devices, and the like;and Industrial devices 128 that can include one or more of motors,pumps, chillers, air compressors, and the like.

In addition, control network 100 can comprise one or more control points106 for receiving user inputs to control each of the one or morecontrollable devices. Control points 106 can be keypads, touch-panels,remote controls, and thermostats. For the purposes of this discussion,and in fulfillment of the dual purposes of clarity and brevity, controlpoints shall herein after be referred to as keypads 106. Additionally,keypads 106 can be user interfaces of the controllable devicesthemselves. Keypads 106 can transmit control commands to and throughcommunication network 102 to control each of the other controllabledevices of control network 100, as well as communicate informationto/from such controllable devices. For example, keypads 106 cancommunicate with each of the controllable devices or with controller 116either directly or via one or more of gateways 114 and/or repeaters 130(repeaters 130 can communicate with additions control networks 100 b,and/or communication networks 102 b, and so on).

According to further aspects of the embodiments, keypad 106 can comprisefeedback indicators to provide feedback to the user. The feedbackindicators can include any combination of visual feedback indicators,haptic feedback indicators, and audible feedback indicators. Feedbackindication control can be provided by keypad 106 upon receiving a userinput, upon requesting feedback, or upon a change in the status of anyof the controllable devices 108-130, and PED 104.

Such controllable lighting devices 110 and controllers 116 can bemanufactured by Crestron Electronics Inc., of Rockleigh, N.J. Forexample, one or more controllable lighting devices 110 and controllers116 can comprise the following devices, each available from CrestronElectronics: CLW-DIMEX wireless lighting dimmer, CLW-DELVEX wirelesslighting dimmer, CLW-SWEX wireless switch, CLW-DIMSWEX wirelessswitch/dimmer combination, CLW-LSWEX wireless lamp switch, CLF-LDIMUEXwireless lamp dimmer, CLWI-DIMUEX universal phase dimmer, CLWI-SWEXin-wall switch, CLWI-1SW2EX in-wall 2-channel switch, CLWI-DIMFLVEX0-10V Dimmer, CLCI-DIMUEX wireless in-ceiling dimmer, CLCI-1 DIMFLV2EXwireless In-Ceiling 0-10V dimmer, CLCI-1SW2EX wireless in-ceilingswitch, CLC-1 DIMFLV2EX-24V wireless in-ceiling 0-10V dimmer.

Other components of control network 100 can also be manufactured byCrestron Electronics Inc. These include one or more controllable shadedevices 112 and controllers 116 that comprise the following devices:CSC-ACEX infiNET EX® Interface to shade motor, CSC-DCEX infiNET EX®interface to Crestron CSM-QMT30 Shades, CSC-DRPEX, and the CSM-QMT50EXQMT motor.

In addition, the one or more keypads 106 can comprise the followingdevices, also available from Crestron Electronics, Inc.: INET-CBDEXCameo® Express Wireless Keypad with infiNET EX®, HTT-B2EXbattery-powered infiNET EX® 2-button Wireless Keypad, and CLWI-KPLEXon-wall wireless lighting keypad.

As described above, sensors 108 can be included in control network 100according to aspects of the embodiments. Such sensors 108 can includeoccupancy sensors, and motion sensors, as well as sensors 108 related tofire and smoke detection, bio-hazard sensors, and the like. The one ormore sensors 108 can comprise the following devices, each available fromCrestron Electronics, Inc. of Rockleigh, N.J.: GLS-OIR-CSM-EX-BATTbattery-powered infiNET EX® occupancy sensor.

Controller 116 can be connected to the various controllable devices viaeither or both of a wired and wireless connection. The one or morecontrollers 116 can be a DIN-AP3MEX DIN Rail 3-Series® AutomationProcessor with infiNET EX®, or an MC3 3-Series Control System® withinfiNET EX®, each of which are available from Crestron Electronics Inc.of Rockleigh N.J. Any one or more of these controllers 116 can provide asubstantially complete integrated automation solution. According toaspects of the embodiments, the various controllable devices of thefacility or enterprise become integrated and accessible throughoperation of controller 116. According to further aspects of theembodiments, controller 116 can be a server, a personal computer (PC),or any other electronic device capable of processing electrical signals.Still further, according to further aspects of the embodiments,controller 116 further comprises a web x-panel project, to allow for PCbased setup. According to still further aspects of the embodiments,controller 116 can be a device manufactured by Crestron Electronics,Inc., of Rockleigh, N.J., comprising a PYNG-HUB. As shown in FIG. 1,controller 116 and gateway 114 can be arranged as two separate devices,but, as indicated by the dashed line forming a box around 114, 116, theycan be arranged to be one device, or contained within a singleenclosure.

According to still further aspects of the embodiments, each of thedevices in FIG. 1 can be interconnected with other components in eitheror both of a wired or wireless manner. For example, PED 104 can connectto internet 132 via a cellular communications interface, or can beconnected through router/firewall device 136 using conventional Ethernetcables. And, as shown in FIG. 1, PED 104 can be connected to LAN 134 viaa wired interface (typically a category 5 type cable, i.e., Ethernetcable), or via a wireless interface such as the Wi-Fi connection that isalso shown. According to still further aspects of the embodiments, eachof the controllable and controlled devices of FIG. 1, such as sensors108, controller 116, and gateway device 114, can use a wireless protocolsuch as infiNET EX. Other wireless communications protocols can also beused.

FIG. 2 is a block diagram of controller 116 for use with control network100 according to an aspect of the embodiments. Controller 116 can beused to control various controllable devices, such as, for example,those described and discussed above that include, among others,controllable devices 108, 110, 112, 118, 120, 122, 124, 126, 128, and130 (security devices (e.g., door locks), lighting system devices,blinds/drapes, HVAC system devices, and sensors such as motion sensors,among many others). One or more controllers 116 can comprise one or morelogic engines for processing control commands.

Controller 116 can include at least one central processing unit (CPU)202, as well as internal bus 220, the operation of which is known tothose of skill in the art. For example, CPU 202 can represent one ormore microprocessors, and the microprocessors may be “general purpose”microprocessors, a combination of general and special purposemicroprocessors, or application specific integrated circuits (ASICs).Additionally, or alternatively, the CPU 191 may include one or morereduced instruction set processors (RISC), video processors, or relatedchip sets. CPU 22 can provide processing capability to execute an, runvarious applications, and/or provide processing for one or more of thetechniques described herein. Applications that can run on controller 116can include, for example, software for processing control commands,software for managing a calendar, software for controlling otherelectronic devices via a control network as noted above, among othertypes of software/applications.

Controller 116 can further include main memory 206, which can becommunicably coupled to CPU 202, and which can store data and executablecode, as known to those of skill in the art. Main memory 206 canrepresent volatile memory such as random access memory (RAM), but mayalso include nonvolatile memory, such as read-only memory (ROM) or Flashmemory. In buffering or caching data related to operations of CPU 202,main memory 206 can store data associated with applications running oncontroller 116.

Controller 116 can also further include nonvolatile storage 204.Nonvolatile storage 204 can represent any suitable, nonvolatile storagemedium, such as a hard disk drive (HDD) or nonvolatile memory, such asflash memory. Being well-suited to long-term storage, nonvolatilestorage 204 can store data files such as media, software, and preferenceinformation. Nonvolatile storage 204 can be removable andinterchangeable, thereby allowing portability of stored files, such asproject files, created during programming of control network 100.According to aspects of the embodiments, project files can be used tomap user desires into functions; as used thusly, project files areconfiguration files. These project files describe all the devicescontrol system 100 knows about, what their buttons are configured to do,what types of devices they are, how they operate, and the operatingparameters, among other features of each controllable device associatedwith control network 100. According to further aspects of theembodiments, project files can also be used to keep track of schedulingdata, which users are using the system (e.g., identifiable by PED 104).

Also shown as part of controller 116 is network interface 208. Networkinterface 208 provides interface capability with one or more of severaldifferent types of network interfaces, including low rate-personal areanetwork (LR-PAN) transceiver 212, wireless local area network (WLAN)transceiver 214, and cellular transceiver 216. Each of transceivers 212,214, and 216 can provide wireless connectivity for controller 116 viarespective ones of first, second, and third antennas 115, 117, and 119.Network interface 208 can represent, for example, one or more networkinterface controllers (NICRs) or a network controller. As those of skillin the art can appreciate, the difference between a LAN and PAN can beless certain, and more one of degree; that is, in some cases, PANs aredefined as those interconnections of devices that are within a fewmeters of each other, while other definitions indicated that devicesthat are within ten meters or so and are interconnected in a manner thatcan be considered to be within a PAN. Regardless of the exactdefinition, or, if no exact definition should ever exist, control system100 can make use of each of a WAN, LAN, and PAN, or sometimes two or allthree at one time, depending on the circumstances, as those of skill inthe art can now appreciate.

According to certain aspects of the embodiments, network interface 208can include LR-WPAN transceiver 212. LR-WPAN transceiver 212 can providecapabilities to network with, for example, a Bluetooth® network, a nearfield communication (NFC) type network, an IEEE 802.15.4 (e.g. ZigBee)network among others. As can be appreciated by those of skill in theart, the networks accessed by LR-WPAN transceiver 212 can, but do notnecessarily, represent low power, low bandwidth, or close range wirelessconnections. LR-WPAN transceiver 212 can permit one electronic device toconnect to another local electronic device via an ad-hoc or peer-to-peerconnection. However, the connection can be disrupted if the separationbetween the two electronic devices exceeds the proscribed rangecapability of PAN interface 212. As those of skill in the art canappreciate, the networks described by IEEE 802.15.4 are mesh-typenetworks, and operate with a central router/coordinator; in controlnetwork 100, the function of such central coordination is performed byone or more controller 116 and/or gateway 114, according to aspects ofthe embodiments.

Network interface 208 can also include WLAN transceiver 214. WLANtransceiver 214 can represent an interface to a wireless LAN, such as an802.11 wireless network. The range of WLAN transceiver 214 can generallyexceed the range available via LR-WPAN transceiver 212. Additionally, inmany cases, a connection between two electronic devices via WLANtransceiver 214 can involve communication through a network router orother intermediary device (not shown in FIG. 2). WLAN transceivers 214can also incorporate an ultra-wideband network.

According to further aspects of the embodiments in regard to controller116, network interfaces 208 can include the capability to connectdirectly to a WAN via cellular transceiver 216. Cellular transceiver 216can permit connection to a cellular data network, such as the enhanceddata rates for global system for mobile communications (GSM) Evolution(EDGE) (also known as enhanced general packet radio service (GPRS)(EGPRS), or international mobile telecommunications (IMT) single carrier(IMT-SC) EDGE network, or other third generation/further generation(3G/4G) cellular telecommunication networks (a detailed discussion ofwhich is both not needed to understand the aspects of the embodiments,and beyond the scope of this discussion). When connected via cellulartransceiver 216, controller 116 can remain connected to the internetand, in some embodiments, to one or more other electronic devices,despite changes in location that might otherwise disrupt connectivityvia LR-WPAN transceiver 212, or WLAN transceiver 214. Also shown in FIG.2 as a component of controller 116 is internal bus 220, which providessignal and data flow to and between all of the internal components ofcontroller 116 in a manner known to those of skill in the art.

As known by those of skill in the art, Ethernet connectivity enablesintegration with IP-controllable devices and allows controller 116 to bepart of a larger managed control network. Whether residing on asensitive, security-conscious corporate LAN 134, a home network, oraccessing Internet 132 through a cable modem, controller 116 can providesecure, reliable interconnectivity with IP-enabled devices, such astouch screens (which can be part of keypad 106), computers, mobiledevices, video displays, Blu-ray Disc® players, media servers, securitysystems, lighting, HVAC, and other equipment—both locally and globally.

Controller 116 can also include one or more wired input/output (I/O)interface 210 for a wired connection between controller 116 and one ormore electronic devices. Wired I/O interface 210 can represent a serialport. A serial port, as those of skill in the art can appreciate, is aserial communication physical interface through which informationtransfers in or out one bit at a time (as opposed to a parallel port,which transmits several bits (typically in groups of 8 bits wide)substantially simultaneously). While it is known that interfaces such asEthernet, FireWire, and USB, all send data as a serial stream, the term“serial port” usually identifies hardware more or less compliant to theRS-232 standard, intended to interface with a modem or with a similarcommunication device.

Wired I/O interface 210 can also represent, for example, a Cresnet port.Cresnet provides a network wiring solution for Crestron keypads,lighting controls, thermostats, and other devices that do not requirethe higher speed of Ethernet. The Cresnet bus offers wiring andconfiguration, carrying bidirectional communication and 24 volt directcurrent (VDC) power to each device over a simple 4-conductor cable.

One or more infrared (IR) interfaces can also be part of wired I/Ointerface 210; the IR interface can enable controller 116 to receiveand/or transmit signals with infrared light. The IR interface can complywith an infrared data acquisition (IrDA) specification for datatransmission. Alternatively, the IR interface can function exclusivelyto receive control signals or to output control signals. The IRinterface can provide a direct connection with one or more devices suchas a centralized AV sources, video displays, and other devices.

Controller 116 can also include, but not necessarily, one or moreprogrammable relay ports 218 a-c. Programmable relay ports 218 can beused by controller 116 to control window shades, projection screens,lifts, power controllers, and other contact-closure actuated equipment.Controller 116 can include, as programmable relay port 218, a“Versiport” relay port that is manufactured by Crestron ElectronicsInc., of Rockleigh, N.J. The Versiport relay port can be managed by aDIN-108 module (also manufactured by Crestron Electronics Inc.), whichis a DIN rail-mounted automation control module that provides eightVersiport I/O ports for interfacing with a wide range of third-partydevices and systems. Each “Versiport” can be configured via software tofunction as a digital or analog sensing input, or as a digital triggeroutput. When configured as a digital input, the Versiport can sense acontact closure or logic level signal from devices such as motiondetectors, partition sensors, alarm panels, 12V triggers, and all typesof switches and relays. When configured as an analog input, theVersiport can sense changes in a resistance or DC voltage level, workingwith everything from temperature and light sensors to water level metersto volume control potentiometers. When operating as a digital output,the Versiport provides a logic level closure signal to trigger controland alarm inputs on a variety of external devices.

Thus, one or more “Versiport” programmable relay ports 218 can enablethe integration of occupancy sensors, power sensors, door switches, orother devices by providing a dry contact closure, low-voltage logic, or0-10 Volt DC signal.

By leveraging remote access of controller 116, a user can control one ormore of the controllable devices and/or environment settings in afacility (home, place of business or manufacture, or enterpriselocation) from substantially anywhere in the world using PED 104. Suchcontrol can be accomplished by a domain name system (DNS) service. Thoseof skill in the art can appreciate that DNS is a hierarchicaldistributed naming system used for computers, services, or any resourcethat is connected to the internet or a private network. According tofurther aspects of the embodiments, controller 116 can be configured toutilize dynamic host communication protocol (DHCP) communications thatinclude a hostname prefixed by a model number. A more detaileddiscussion of the internal operation of controller 116 is not needed tounderstand the various aspects of the embodiments described herein, andtherefore is beyond the scope of discussion herein. However, suchdetailed discussion can be found in the aforementioned Applicants'co-pending U.S. Non-Provisional Patent Application, as referenced above.

According to aspects of the embodiments, controller 116 hosts a projectfile, such as a Crestron Core 3 project file, also referred to as “SmartGraphics [Project],” which is intended to be used by one or more mobiledevices (such as PED 104) with a control application (App) such as aCrestron App (located on PED 104). As described above, one or moreproject files can be created during the installation of control network100. The Crestron App is designed to receive and render the SmartGraphics project file. The Crestron App is responsible for communicatingtaps and feedback to the user. Additionally, Smart object App can becreated for use with a local Crestron Mobile Pro Project as well as withforeign AV processors. The Crestron Mobile Pro project can contain justa Core 3 Smart Object and nothing else. The Smart Graphics Project file,located on controller 116, is a collection of items that are meaningfulin some way to a control system program, such as Crestron App. Thiscollection of items can include things like “buttons,” “sliders,” or“text” (among other graphical representations). According to furtheraspects of the embodiments, Smart graphics project file can include“smart object” file, which can be a predefined conglomeration of otherobjects (buttons, slides, among others). For example, a lighting smartobject file can comprise a slider to report/set a light level, and a fewbuttons to raise/lower and turn on/off the lights. According to furtheraspects of the embodiments, in control network 100, smart object filetalks directly to a Pyng-HUB, such as controller 116. As such, smartobject files can be used in or by any smart graphics project file, andthey'll communicate with control network 100 and Crestron App, even ifthe project (i.e., the program currently being executed) is intended tocontrol an external AV processor.

Referring back to FIG. 1, control network 100 further comprisescommunication network 102 that provides access with and between devicesof control network 100 according to aspects of the embodiments.Communication network 102 can be a PAN, LAN, metropolitan area network,WAN, an alternate network configuration, or some other combination ofnetwork types and/or topologies.

According to an aspect of the embodiments, communication network 102 canemploy both wired and wireless communication protocols. For example, thecontrollable devices can form communication network 102 with gatewaydevice 114 (operating in a wireless manner) by communicating over ashort range communication protocol such as Crestron infiNET EX wirelessprotocol (e.g., the IEEE 802.15.4 wireless protocol). Or, according to adifferent aspect of the embodiments, gateway device 114, operating in awired manner, can form a LAN with PED 104 communicating via Ethernetprotocols using a wire-based Ethernet capability (it can also do so in awireless manner). According to a further aspect of the embodiments,controller 116 or PED 104 can connect via a WAN such as the world wideweb to access data stored on a remote server (not shown in FIG. 1).

According to further aspects of the embodiments, communication network102 can be a public switched telephone network (PSTN). Alternatively,communication network 102 can further include a cable telephony network,an internet protocol (IP) telephony network, a wireless network, ahybrid cable/PSTN network, a hybrid IP/PSTN network, a hybridwireless/PSTN network, or any other suitable communication network 102or combination of communication networks. In addition, other networkembodiments can be deployed with many variations in the number and typeof devices, communication networks, the communication protocols, systemtopologies, and myriad other details without departing from the spiritand scope of the aspects of the embodiments.

Referring now to FIG. 3, control network 100 can include one or moregateway devices 114. According to a further aspect of the embodiments,controller 116 further comprises a built-in gateway 114. According tostill further aspects, control network 100 can comprise an externalgateway 114, such as a CEN-RFGW-EX gateway, available from CrestronElectronics, Inc.

According to aspects of the embodiments, gateway 114 of control network100 provides network devices with an entrance to communication network102 through controller 116 and can include software and/or hardwarecomponents to manage traffic entering and exiting communication network102 and conversion between the communication protocols used by thenetwork devices and communication network 102.

Gateway 114 can be configured to operate in both a wired a wirelessmanner and act as the network coordinator, and can further managenetwork configurations. Additionally, gateway 114 can be configured tocommunicate with controller 116 via wired I/O interface 210, such as anEthernet interface (IEEE 802.3). One such gateway 114 according to anaspect of the embodiments is the CEN-RFGW-EX wireless gatewaymanufactured by Crestron Electronics, Inc., and which is a two-way radiofrequency (RF) gateway\transceiver designed to enable communications andmanagement for a complete infiNET EX wireless network of dimmers,keypads, remote control devices (RCDs), among other types of devices.The CEN-RFGW-EX wireless gateway links the infiNET EX network to aCrestron control system via a wired connection such as Ethernet orCresnet. infiNET EX dimmers, switches, keypads, thermostats, and otherdevices, can be linked to controller 116 via a single CEN-RFGW-EXgateway 114. Additional gateways 114 can be installed to support moredevices. Wireless expanders (not shown in FIG. 1) can be added whereverneeded to extend control network 100 by filling in gaps between devices.That is, according to aspects of the embodiments, expanders canreinforce the network when operating in accordance with mesh networksprinciples.

FIG. 3 illustrates a block diagram of gateway 114 according to an aspectof the embodiments. Gateway 114 can include one or more transceivers212, 214, and 216, which can provide connectivity for gateway 114 whenacting in a wireless manner. In addition to the transceivers 212, 214,216, gateway 114 can further include a network interface (NWI) thatcomprises one or more network interface cards (NICs), or networkcontrollers (NWCs). In certain embodiments, the network interface caninclude LR-WPAN transceiver 212, which can provide capabilities tonetwork with, for example, a Bluetooth® network, NFC network, or aZigBee/Infinet network, among others. As can be appreciated by those ofskill in the art, the networks accessed by LR-WPAN transceiver 212 can,but do not necessarily, represent low power, low bandwidth, or closerange wireless connections, such as that used by second antenna 117.LR-WPAN transceiver 212 can permit one electronic device to connect toanother local electronic device via an ad-hoc or peer-to-peerconnection.

Gateway 114 can further include wired I/O interface 210, which canrepresent an interface to a wired Ethernet-based network. Gateway 114includes WLAN transceiver 214, which can access an IEEE 802.11x wirelessnetwork. The range of the WLAN interface (WLAN transceiver 214) cangenerally exceed the range available via the PAN interface.Additionally, in many cases, a connection between two electronic devicesvia the LAN interface can involve communication through a network routeror other intermediary devices. As discussed above, gateway 114 furthercomprises LR-WPAN transceiver 212 that can access an IEEE 802.15.4 (e.g.ZigBee/InfiNet) network. As those of skill in the art can appreciate,the networks described by IEEE 802.15.4 are mesh-type networks, andoperate with a central router/coordinator; in control network 100, thefunction of such central coordination is performed by one or morecontroller 116 and/or gateway 114, according to aspects of theembodiments.

In a wired configuration, wired I/O interface 210 can be a LANpower-over-Ethernet (PoE) interface that can be fashioned using an8-wire RJ-45 female connection with two LED indicators. According to afurther aspect of the embodiments, a another type of NWI can be Cresnetinterface 302 b, which is a 4-pin 3.5 millimeter (mm) detachableterminal block providing an interface for Cresnet proprietarycommunications on a LAN that includes power-over-Ethernet (PoE). The PoEinterface can be configured for receiving both an electric power signaland an information signal from a control network. For example, Cresnetinterface 302 b can be connected through category 5 cable (CAT 5) to aLAN that contains a power supply, multiple control points, and signalgenerators. Through Crestnet interface/LAN PoE interface 302 b, gateway114 can interface with control network 100. For example, gateway 114(which can be both wired and wireless) can communicate with controller116, such as a PRO3 available from Crestron Electronics, Inc.

Gateway 114 comprises one or more connectors, indicators and interfacebuttons, as well as an antenna connection for the supplied antenna.Gateway further comprises LED indicators, such as power on/off LED 304,network activity indicator LED 306, and activity indicator LED 308.Power on/off LED 304 is an indicator that shows that operating power isbeing supplied to gateway 114 whether from the Cresnet network or a PoEconnection. Network LED indicator 306 shows that communication with theCresnet system is occurring. Activity indicator LED 308 shows thatwireless communications are occurring, such as those that involve theelements of the wireless PAN.

Gateway 114 further comprises acquire button 310 and setup button 312.Acquire button 310 and setup button 312 can be recessed push buttonseach with an indicator LED. Acquire button 310 can be employed toconfigure communication with the PAN and setup button 312 can beemployed to configure communication with control network 100.

Gateway 114 can be placed in the “Acquire” mode via acquire button 310or a different means, such as the pushing of certain buttons in acertain order. The associated LED can indicate that gateway 114 is inthe “Acquire” mode. Once gateway 114 has been placed in the “Acquire”mode, the joining device can be brought into range and can be placed inthe “Acquire” mode to be acquired by gateway 114 through a certainsequence. Such sequence involves the pushing of buttons in a certain,specific order, a detailed discussion of which has been omitted infulfillment of the dual purposes of clarity and brevity. By pushingacquire button 310 a second time (within a predetermined time period),gateway 114 can exit the “Acquire” mode as indicated by theLED-illuminated acquire button 310.

As discussed above, control network 100 can further comprise PED 104.PED 104 can be a smart phone, tablet, remote control, personal digitalassistant (PDA), or any other electronic device configured forpresenting a user interface, such as a graphical user interface (GUI)and receiving user inputs, such as in the form of selections from agraphic user interface.

FIG. 4 illustrates a block diagram of a personal electronic device 104for use with control system 100 according to aspects of the embodiment.PED 104 can include at least one central processing unit (CPU) 402. Forexample, CPU 402 can represent one or more microprocessors, and themicroprocessors can be “general purpose” microprocessors, a combinationof general and special purpose microprocessors, or ASICs. Additionally,or alternatively, CPU 402 can include one or more reduced instructionset (RISC), advanced RISC machine (ARM), or complex instruction set(CISC) processors, video processors, or related chip sets. CPU 402 canprovide processing capability to execute an operating system (OS), runvarious applications, and/or provide processing for one or more of thetechniques described herein. Applications that can run on PED 104 caninclude, for example, software for managing and playing AV content,software for managing a calendar, software for controlling telephonecapabilities, software for controlling other electronic devices via acontrol network as noted above, as well as software for controllingvarious other functions and interconnected devices.

PED 104 further comprises main memory 414, which can be communicablycoupled to CPU 402, and which may store data and executable code. Mainmemory 414 can represent volatile memory such as RAM, but can alsoinclude nonvolatile memory, such as ROM or flash memory. In buffering orcaching data related to operations of CPU 402, main memory 414 can storedata associated with applications running on PED 104.

PED 104 can also include nonvolatile storage 412. Nonvolatile storage412 can represent any suitable nonvolatile storage medium, such as a HDDor nonvolatile memory, such as flash memory. Being well-suited tolong-term storage, nonvolatile storage 412 may store data files such asmedia, software and preference information. Nonvolatile storage 412 canbe removable and interchangeable, thereby allowing portability of storedfiles such as project files created during programming of controlnetwork 100. Those of skill in the art can appreciate that dataassociated with controlling certain other electronic devices, such as aproject file for a control application, can be saved in nonvolatilestorage 412.

Display 410 can display images and data for PED 104. As those of skillin the art can appreciate, display 410 is optional. If included in PED104, however, display 410 can use any type of display technology, suchas, but not limited to, a liquid crystal display (LCD), a light emittingdiode (LED) based display, an organic light emitting diode (OLED) baseddisplay, a cathode ray tube (CRT) display, or an analog or digitaltelevision, among other types. According to other aspects of theembodiments, display 410 can function as a touch screen display throughwhich a user can interact with PED 104.

PED 104 can further include user interface 408. User interface 408 caninclude indicator lights and user input structures, but can also includea GUI on display 410. As those of skill in the art can appreciate, userinterface 408 can operate via CPU 402, using memory from main memory 414and long-term storage in nonvolatile storage 412, among using othertypes of memory (such as an HDD, not shown in FIG. 4). According toaspects of the embodiments, if display 410 is not included in PED 104,indicator lights, sound devices, buttons, and other various input/output(I/O) devices can allow a user to interface with PED 104. If, however,display 410 is included in PED 104 and uses a GUI, user interface 408can provide interaction with interface elements on display 410 viacertain user input structures, user input peripherals such as a keyboardor mouse, or a touch sensitive implementation of display 410.

As can be appreciated by those of skill in the art, one or moreapplications can be opened and accessible to a user via user interface408 and displayed on display 410 of PED 104. One or more of the openedapplications can be run on CPU 402 in conjunction with main memory 414,nonvolatile storage 412, display 410, and user interface 408.Instructions stored in main memory 414, nonvolatile storage 412, or CPU402 (CPU 402 can have its own internal storage, of many differenttypes), of PED 104 can enable a user to install control network 100. Assuch, those of skill in the art can appreciate that the instructions forcarrying out such techniques on PED 104 can represent a standaloneapplication, a function of the OS on PED 104, or a function of thehardware of CPU 402, main memory 414, nonvolatile storage 412, or otherhardware of PED 104.

One such application that can be opened and accessible to the user is aconfiguration application for installing control network 100 accordingto an aspect of the embodiments. The configuration application can bedownloaded from an application marketplace such as from the Google Playapplication marketplace or the Apple iTunes® application marketplace,among other market places available through the internet, or othernetworks.

As briefly described above, the project file provides the instructionsallowing the control application to communicate with the target controlnetwork (control network 100, according to aspects of the embodiments).Further, the project file comprises the menu pages of the controlapplication corresponding to the locations of controllable devices. Forexample, the control application can display one or more menu pagesidentified by page identities for controlling the one or morecontrollable devices on control network 100 according to the projectfile. The menu pages comprise selectable elements corresponding tocontrol functions as defined in the project file.

The configuration application displays a series of menu pages comprisingselectable elements and graphical elements. As will be described ingreater detail below, the one or more of the selectable elements cancorrespond to initialization functions of the configuration application.PED 104 can transmit signals to control network 100 according to theinitialization functions selected by the user. Additionally, controlnetwork 100 can communicate with PED 104, such as by providing feedbacksignals to PED 104. According to an aspect of the embodiments, PED 104can communicate with controller 116 running a logic engine viacommunication network 102. Gateway 114, according to further aspects ofthe embodiments, can be used to relay commands and return statusinformation to and from sensors 108 and from the various controllabledevices 110, 112, 118, 120, 122, 124, 126, 128, and 130

According to various aspects of the embodiments, PED 104 can includelocation sensing circuitry 406. Location sensing circuitry 406 cancomprise global positioning system (GPS) circuitry, but can alsorepresent one or more algorithms and databases, stored in nonvolatilestorage 412 or main memory 414 and executed by CPU 402, which may beused to infer the location of PED 104 based on various observed factors.For example, location sensing circuitry 406 can represent an algorithmand database used to approximate geographic location based on thedetection of local 802.11x (Wi-Fi) networks or nearby cellular phonetowers.

PED 104 can also include wired I/O interface 210 for a wiredinterconnection between a first electronic device and a secondelectronic device. Wired I/O interface 210 can represent, for example, auniversal serial bus (USB) port, an IEEE 1394 port, or a FireWire® port.However, wired I/O interface 210 can also represent a proprietaryconnection. Additionally, wired I/O interface 210 interface can permit aconnection to user input peripheral devices, such as a keyboard or amouse. In addition to wired I/O interface 210, PED 104 further comprisesIR interface 430 that can enable PED 104 to receive and/or transmitsignals with infrared light. By way of example, IR interface 430 cancomply with an infrared IRDA specification for data transmission.

One or more network interfaces 208 can also be provided in PED 104. Oneor more of such network interfaces 208 can provide additionalconnectivity for PED 104. Network interfaces 208 can represent, forexample, one or more NICs or a network controller. In certainembodiments, the network interface 208 can include LR-WPAN transceiver212. LR-WPAN transceiver 212 can provide capabilities to network with,for example, a Bluetooth® network, an NFC network, or a ZigBee/CresNetnetwork. As should be appreciated, the networks accessed by LR-WPANtransceiver 212 can, but do not necessarily, represent low-power,low-bandwidth, or close range wireless connections. However, as those ofskill in the art can appreciate, the connection in a PAN can bedisrupted if the separation between the two electronic devices exceedsthe operational range capability of LR-WPAN transceiver 212. LR-WPANtransceiver 212 can permit one electronic device to connect to anotherlocal electronic device via an ad-hoc, or peer-to-peer connection, suchas that defined by the wireless PAN protocol IEEE 802.15.n,communications network 102.

LR-WPAN transceiver 212 can also incorporate IEEE 802.15.4 (e.g. ZigBee)network, or an ultra-wideband network. As those of skill in the art canappreciate, the networks described by IEEE 802.15.4 are mesh-typenetworks, and operate with a central router/coordinator; in controlnetwork 100, the function of such central coordination is performed byeither or both of controller 116 and/or gateway 114, according toaspects of the embodiments.

Network interface 208 can also include WLAN transceiver 214. WLANtransceiver 214 can represent an interface to a wireless LAN, such as anIEEE 802.11x wireless network (Wi-Fi). The wireless operating rangecapability of LAN interface 426 can generally exceed the wirelessoperating range capability available via LR-WPAN transceiver 212.Additionally, in many cases, a connection between two electronic devicesvia WLAN transceiver 214 can involve communication through a networkrouter or other intermediary devices. In PED 104 WLAN transceiver 214interfaces with first antenna 115, WLAN transceiver 212 interfaces withsecond antenna 117, and cellular transceiver 216 interfaces with thirdantenna 119 according to aspects of the embodiments. Communicationsnetwork 134 is a wired or wireless LAN, such as that defined by IEEE802.11.n (Wi-Fi), or 802.3 (Ethernet).

According to further aspects of the embodiments, network interfaces 208of PED 104 can further include the capability to connect directly to aWWAN via cellular transceiver 216, and third antenna 119 according toaspects of the embodiments. Cellular transceiver 216 can permit aconnection to a cellular data network, such as an EDGE network, oranother 3G/4G network, among others. When connected via cellulartransceiver 216, PED 104 can remain connected to the internet and, insome embodiments, to other electronic devices, despite changes inlocation that might otherwise disrupt connectivity via LR-WPANtransceiver 212, or WLAN transceiver 214. As will be discussed ingreater detail below, wired I/O interface 210 and network interfaces 208can represent both low- and high-bandwidth communication channels fortransferring user data using the simplified data transfer techniquesdiscussed herein.

PED 104 can also include NFC interface 416. NFC interface 416 can allowfor extremely close range communications at relatively low data rates(e.g., about 464 kilo-bits/second (kb/s)), and can comply with suchstandards as International Standards Organization (ISO) 18092 or ISO21521, or it can allow for close range communications at relatively highdata rates (e.g., about 560 mega-bits/second (Mb/s)), and can complywith the TransferJet® protocol. NFC interface 416 can have a range ofbetween about 2 to about 4 centimeters (cm) (or between about 0.78″ toabout 1.57″). The close range communication with NFC interface 416 cantake place via magnetic field induction, allowing NFC interface 416 tocommunicate with other NFC interfaces, or to retrieve information fromtags having radio frequency identification (RFID) circuitry. Asdiscussed in greater detail below, NFC interface 416 can provide amanner of initiating or facilitating a transfer of user data from oneelectronic device to another electronic device.

PED 104 can also include camera 420. With camera 420, PED 104 can obtaindigital images or videos. In combination with optical characterrecognition (OCR) software, barcode-reading software, ormatrix-code-reading software running on PED 104, camera 420 can be usedto input data from printed materials having text or barcode information.Such data can include information indicating how to control anotherdevice from a matrix barcode that can be printed on the other device, asdescribed in greater detail below.

According to further aspects of the embodiments, PED 104 can alsoinclude one or more accelerometers 418 that can sense the movement ororientation of PED 104. Accelerometers 418 can provide input or feedbackregarding the position of PED 104 to certain applications running on CPU402. According to further aspects of the embodiments, accelerometer 418can be provided by devices made using microelectromechanical system(MEMS) technology. MEMS devices, which can be defined as die-levelcomponents of first-level packaging, can include pressure sensors,accelerometers, gyroscopes, microphones, digital mirror displays,microfluidic devices, among other devices.

According to aspects of the embodiments, control network 100 can beconfigured to be installed by untrained users executing a configurationapplication on PED 104. According to further aspects of the embodiments,the control system and associated configuration application are referredto as Pyng, which are software programs created and manufactured byCrestron Electronics, Inc., of Rockleigh, N.J.

One such application that can be opened and accessible to the user is aconfiguration application for installing control network 100 accordingto an aspect of the embodiments. The configuration application can bedownloaded from an application marketplace such as from the Google Playapplication marketplace, or the Apple iTunes® application marketplace,among other application market places available through the internet, orother networks. A detailed discussion of the configuration applicationis both not needed to appreciate the various aspects of the embodiments,and can be found in the co-pending U.S. Non-provisional PatentApplication referenced above; as such, a detailed discussion has beenomitted in fulfillment of the dual purposes of clarity and brevity.

Attention is now directed towards FIG. 5, which illustrates wall mountkeypad (keypad) 106 that can be used in control network 100 as part ofan acoustic sensory network (ASN) according to aspects of theembodiments. Keypad 106 includes display/touch panel 502 (an interactivedisplay that can be an LCD or LED, or combination thereof),microphone(s) 504, optional audio processing board 505 (which comprisespre-amplifier 506, analog-to-digital converter (ADC) 508, and 60 Hertz(Hz) notch filter according to an aspect of the embodiments), processor512, IEEE 811.15.4 LR-WPAN transceiver (transceiver) 212, WLANtransceiver 214, cellular transceiver 216 (and their respectiveantennas, 115, 117, and 119), internal bus 516, antenna 117,LAN/Ethernet connector 518, and voice recognition (VR) system-on-a-chip(SoC) circuit (VR SoC circuit) 520. As those of skill in the art canappreciate, other components have been omitted from FIG. 5 infulfillment of the dual purposes of clarity and brevity, as they wouldnot aid in understanding the various aspects of the embodiments.According to further aspects of the embodiments, the devices, software,algorithms, and other components of the ASN are described in referenceto keypad 106 and controller 116, but can be distributed in one or moreof any of the devices of network 100, e.g., gateway 114. In addition,one or more of the components of the ASN can be separated from keypad106, such as microphones 504, audio processing board 505, and VR SoCcircuit 520, according to aspects of the embodiments. For example,either or both of audio processing board 505 (and its components) or VRSoC circuit 520 can be included in controller 116 or gateway 114.However, in fulfillment of the dual purposes of clarity and brevity, andaccording to aspects of the embodiments, the following description ofthe devices, software, algorithms, and other components of the ASN shallbe made in reference to keypad 106 and controller 116.

The ASN according to aspects of the embodiments includes audioprocessing components to interpret spoken words as commands to controlthe controllable devices, including those of lights and related devices,in such a manner as to overcome the problems of the prior art aspreviously described. That is, the ASN can include one or moremicrophones 504 a,b, VR SoC circuit 520, or, in the alternative,optional audio processing board 505, to capture, process, and implementaudible commands. As those of skill in the art can appreciate, VR SoCcircuit 520 contains the necessary components to convert the audiosignals received by each mic 504 a,b, into digital form, providefiltering before and/or after conversion to digital form, performadditional processing (as described in greater detail below), and can,according to aspects of the embodiments, include software to identifythe word (or words) that were spoken to produce the digital words. Suchprocessing can be referred to as acoustic finger printing, or voicerecognition. Further, VR SoC circuit 520 can also provide a time stampto the received audio signal, which can be further used in processing ina manner to be described below, or the time stamp can be provided byother circuitry, such as, for example, controller 116. Optionally,substantially similar processing can occur in audio processing board505; however, in fulfillment of the dual purposes of clarity andbrevity, discussion shall only be made in regard to VR SoC circuit 520.

According to still further aspects of the embodiments, either or both ofVR SoC 520 and audio processing board 505 can be implemented in one ormore of the dimmers, wall mounted touch panels, remote control devices,and the like, all of which can be considered to be part of controlnetwork 100 and the ASN according to aspects of the embodiments.

In cases where it is implemented, audio processing board 505 accepts asan input the analog audible signal (audible signal) received from eachof microphones 504 a,b, applies a pre-amplification to the signal toscale it, then converts the same to a digital audible signal using ADC508. The “raw” output of ADC 508 can then be filtered by notch filter510 to remove as much 60 Hz “hum” as possible, and the filtered digitalaudible signal (digital audible signal) can then be sent to processor512 for further processing.

Attention is also directed to FIG. 10, which illustrates several audioprocessing blocks that can occur within either or both of audioprocessing board 505 and VR SoC circuit 520 according to aspects of theembodiments. In fulfillment of the dual purposes of clarity and brevity,however, reference will be made as to the processing blocks as occurringwithin VR SoC circuit 520, although that need not necessarily be thecase; one or more of the processing blocks shown is within VR SoCcircuit 520 can also be implemented in one or more separate devices,such as ASICs, or even the processors of keypad 106, controller 116, andkeypad 104, among other devices of network 100.

Referring now to FIGS. 5 and 10, microphones 504 a,b receive audiocommands spoken by a user or occupant of the home or enterpriselocation, along with other extraneous audio signals, the latter of whichcan be collectively referred to as audio noise; thus, the combinedanalog audio signal consists of an audio command and audio noise, and isrepresented as the analog audio signal from microphone 504 a, or AASa,and similarly from mic 504 b, AASb. Within VR SoC circuit 520, both ofthe analog microphone output signals encounter analog processing circuit1002 a,b respectively. Analog processing circuit can include, amongother circuitry, 60 Hz notch filters, one or more of a low pass filter,high pass filter and a bandpass filter, pre- and post-amplifiers, and anADC. The output of analog processing circuits 1002 a,b can be referredto as digital audio signal “a” and “b,” respectively, DASa, DASb.Although some noise has been removed, DASa,b both still contain ambient,background noise that can include one or more noise signals generated byfans, motors, audio sources, and non-command words, among others.

According to further aspects of the embodiments, upon conversion from ananalog to digital form, each of DASa,b can have a time-date stampapplied to it. According to aspects of the embodiments, the applicationof a time-date stamp can be applied to the received digital audio signalby time-date stamp generator 1004. As those of skill in the art canappreciate, any processing steps that occur at processor 512 of keypad106 (or any other similarly situated control or controllable device ofcontrol network 100) can also occur in one or both of gateway 114 andcontroller 116. The time-date stamp applied by time-date stamp generator1004 can be used one or more different ways. For example, because eachmicrophone's output can be time-stamped, it can be determined whichmicrophone was closest to the source of the audio command (onceprocessing occurs to decipher the command, discussed below). Further,because in subsequent processing the two microphones' outputs arecombined, a single average time stamp value can be generated and appliedto the combined audio command output from the respective keypad 106;this average time stamp can then be used and compared to other timestamps generated by other keypads 106 to further ascertain which keypadfirst received the audio command.

There are several mechanisms through which time stamps can be generatedand applied according to aspects of the embodiments. As those of skillin the art can appreciate, a certain degree of accuracy is required ingenerating the time stamp in order to make the time stamp useful indetermining the order of arrival of audio signals. The speed of sound,V_(s), is about 1126 feet-per-second (fps) at sea level, under certainpredefined conditions. The speed varies with temperature, humidity, andaltitude, as those of skill in the art can appreciate, but the generallyaccepted “norm” value of V_(s) of 1126 fps can be used for the purposesof this discussion. Using this value yields a travel time of about 888.1microseconds-per-foot (μs/ft). Most clock speeds of the processors andother digital circuitry will operate at much higher frequencies than1100 Hz; however, it is not the absolute clock frequency that isimportant (though below a certain value, as those of skill in the artcan appreciate, time stamping would not be effective), but that theclock speeds be substantially identical at each keypad 106, and besubstantially in synchronization with each other, at least by severalorders of magnitude in regard to the expected differences in timebetween when a first keypad marks the audio sound and a second keypadmarks the same audio sound. According to aspects of the embodiments,there are several network communication protocols that can be used togenerate time stamps of sufficient accuracy; one such system usesZigBee, as described in papers entitled “Time Synchronization for ZigBeeNetworks,” Cox, D. et al., IEEE 0-7803-8808, September, 2005, and“Non-invasive Time Synchronization for ZigBee Wireless Sensor Networks,”Ferrari, P., et al., IEEE 978-1-4244-2275, March, 2008, the entirecontents of both of which are incorporated herein by reference.Time-date stamp generator 1004 uses either or both the protocolsdescribed above, among others not mentioned, and generates the time-datestamp that is then added to digital word represented the amplitudeoutput from each respective microphone; in this manner, the digital wordnow resembles a packet of data familiar to those of skill in the art ofdigital data transmission using protocols such as the Open SourceInitiative (OSI) model and internet protocols, among others.

According to still further aspects of the embodiments, application ofthe time-date stamp can occur in one or both of controller 116 orgateway 114 after the audible command has been received from each of thekeypads, assigned an identifier header (or footer), and transmittedfollowing digitization. Thus, the digitized audible commands can be sentin real time to controller 116 or gateway 114. Considering thesubstantially instantaneous rate of communication from each keypad tothe controller 116 (for purposes of this discussion, the use ofcontroller 116 alone will be considered by means of a non-limitingexample), such delay in time-date stamping can be negligible. Accordingto further aspects of the embodiments, test communications can be sentto each keypad in order to ascertain a trip delay, and such delay timecan be subtracted from each received audible command digital word afterit has been received and time-date stamped, and prior to processing, asdescribed below in regard to FIG. 8, and method 800.

Following the application of the time-date stamp by time-date stampgenerator 1004, acoustic echo cancellation (AEC) algorithms can beapplied through use of AEC processors 1006 a,b to the respectivedigitals signals, DASa,b. The implementation and use of AEC processingis known to those of skill in the art, and therefore, in fulfillment ofthe dual purposes of clarity and brevity, a detailed discussion thereofneed not be repeated herein. However, one known goal of use of AEC is toreduce extraneous media sounds from the digital audio signals. As thoseof skill in the art can appreciate, AEC can use an audio signal as areference, and then cancels this reference signal from the microphoneinput. The reference signal can be provided to each AEC circuit from oneor both of controller 116 and gateway 114. Such reference signal can bethe audio portion of any video that might be playing in each respectiveroom of the corresponding keypad 106, or audio signal that it beingprovided to amplifiers and speakers in each corresponding room. The AECcircuit uses a Least Mean Square (LMS) algorithm to create an adaptivefilter used to eliminate the reference and acoustic echoes associatewith it. In addition, non-linear adaptive filtering can also be used tofurther suppress this signal. According to aspects of the embodiments,this processing can occur in keypad 106, and the reference signal willthen be provided to each keypad 106 from the media system to therespective keypad 106 either through the IEEE 802.15.4 radio connection(LR-WPAN transceiver 212 and second antenna 117), IEEE 802.11 radioconnection (WLAN transceiver 214 and first antenna 115), cellulartransceiver 216 (and third antenna 119), or via the IEEE 802.3 LANconnection (wired I/O interface 210). According to further aspects ofthe embodiments, AEC blocks 1006 a,b can also include reverb reductionand/or active noise cancellation, the operation of which are known tothose of skill in the art.

Following AEC in AEC blocks 1006 a,b, the digital audio signal can befurther processed and/or enhanced according to several further aspectsof the embodiments. Each DAS from the respective microphones 504 a,b ina first keypad 104 a can have direction of arrival (DOA) processingperformed in conjunction with directionally adapted beamforming (DABF);this processing can occur in DOA block 1008, which, as shown in FIG. 10,comprises at least two inputs, the respective DASs from each microphone504 a,b in first keypad 104 a. The output of DOA block 1008 is anotherdigital packet of data that includes the DOA processed digital audiosignals and a relative angle that each microphone 504 a,b received itsanalog audio signal. The DABF block 1010 receives that output from DOAblock 1008 and uses the directional information (the relative angle) andfurther reduces the noise in the portion of the digital packet thatrepresents the digitized and further processed audio signals. In thismanner, the output of VR SoC circuit 520 has now substantially minimizedor reduced the noise that accompanied the spoken command, so that thesignal-to-noise ratio (SNR) has been improved of the spoken audiocommand.

A detailed discussion of the processing that occurs in either or both ofDOA block 1008 and DABF block 1010 is not necessary to the understandingof the aspects of the embodiments; however, such processing is describedin the following documents, the entire contents of each of which areincorporated herein in their entirety. Such documents include “A New DOAEstimation Method Using a Circular Microphone Array,” Karbasi, A., etal., School of Comp. and Commun. Sciences, Ecole PolytechniqueF'ed'erale de Lausanne CH-1015 Lausanne, Switzerland, and CommonPlatform Software Research Labs., NEC Corporation Kawasaki 211-8666,Japan, 2007; “Direction of Arrival Estimation Using the ParameterizedSpatial Correlation Matrix,” Dmochowski, J., et al., IEEE TransactionsOn Audio, Speech, and Language Processing, Vol. 15, No. 4, May 2007; and“Microphone Arrays: A Tutorial,” McCowan, I., derived from “RobustSpeech Recognition using Microphone Arrays,” McCowan, I., PhD Thesis,Queensland University of Technology, Australia, 2001.

In addition, the following websites provide further information as toimplementation of direction of arrival and directionally adaptivebeamforming circuitry and processing:https://www.xmos.com/support/boards?product=35564, andhttp://www.vocal.com/voice/, both of which were current and available asof the date of filing of this U.S. Non-provisional Patent Application,and the entire contents of each of which are incorporated herein intheir entirety.

According to aspects of the embodiments, DOA and DABF processing can beimplemented at each keypad 106 in regard to the one or more microphones504 in the respective keypad 106, or, DOA and DABF processing can occurat a central location, such as in controller 116 and CPU 202 (or ingateway 114, among other “central locations”). Or, according to stillfurther aspects of the embodiments, DOA processing can occur at eachkeypad, and DABF processing can occur at the central location such ascontroller 116 and CPU 202. Further, while it has been discussed anddescribed that keypad 106 can have two microphones 504, according tofurther aspects of the embodiments, keypads 106 can have one, three, oreven more microphones 504. Further still, one or more such microphones504 can be stand alone units, i.e., one or more or a plurality of justmicrophones can be installed in one or more of the rooms/hallways of thehome or enterprise location without keypad 106 in order to provide alarger area to listen for commands, and obtain more detailed spatialinformation about the location of the source of the audible command, aswell as increasing the likelihood of accurately determining the room towhich the command is being directed towards.

Following the processes described above, the digital audible signal canbe processed by a speech recognition algorithm in order to attempt todiscern the command that is contained in the digital audible signal.Such processing can be performed in each keypad 106. According tofurther aspects of the embodiments, however, such processing shall bediscussed from the perspective of occurring in controller 116, withinCPU 202, in fulfillment of the dual purposes of clarity and brevity.However, those of skill in the art can appreciate that such any of thisprocessing, as well as additional processing described and discussedabove and below, can be distributed throughout a network such as network100, and could occur, for example, in one or more of gateway 114.

For example, one such command could be “lights off,” as the person isleaving a bathroom. In a first scenario, it will be presumed that noextraneous noise exists, and that the command can only be heard by thekeypad/processor in the room to which it is directed. Then, using aspeech recognition algorithm in the respective keypad 106, processor 512could relatively easily act on the command, and turn off the lights inthe bathroom. Such “turning off” command can be acted upon even if thesame command was received by an adjoining bedroom because of therelatively high percentage of certainty of the true nature of thecommand contained in the digital audible signal. As those of skill inthe art can appreciate, speech recognition capabilities can be locatedthroughout a residential or commercial facility to facilitate control ofdevices in the residence/office/enterprise location. However, as hasbeen discussed above, extraneous noise does exist and it cannot be saidwith any degree of certainty that processor 512 of keypad 106 in thebathroom would act on such command, or that the lights of the bedroomnext door would not also be turned off, much to the surprise of theoccupant therein.

Therefore, according to further aspects of the embodiments, furtherprocessing of the received audio command can also determine theamplitude of the received digital audible signal at each respectivekeypad 106; such amplitude can be used by CPU 202 in controller 116 (oranother processor, such as processor 512, though discussion will bedirected to such processing occurring within CPU 202 from hereon in), tocompare the amplitude of a plurality of received digital audible signalsDASN. Relative amplitude between all of the received signals can be usedto assist in determining which of the received signals was receivedfirst, as amplitude falls off with distance, as time increases as well.Therefore, if one of the received audio signals has a larger amplitudethan the other, then the microphone associated with the (digital)audible signal with the larger amplitude can be considered to be thedevice to which the command is directed towards, or at least it can beconsidered as a factor to take into account. FIGS. 6 and 7, which aredescribed in greater detail below, illustrate the principles ofoperation of the time-date stamp and amplitude determination.

According to further aspects of the embodiments, either or both ofadditional processing and additional circuitry can be used that reducesthe likelihood of misinterpreting the received digital audio signal. Thefirst item to be considered is the use of two microphones 504 a,b ineach keypad 106, as shown in FIG. 5. The outputs of mics 504 a,b areboth directed to VR SoC circuit 520, or audio processing board 505; asdiscussed above, only use of the former will be described herein infulfillment of the dual purposes of clarity and brevity. When two (ormore) mics 504 a,b are used, the effects of noise on the intended audiosignal can be reduced. Some sources of noise can include other peoplespeaking, fans in bedrooms or bathrooms, ceiling speakers, televisions,cell-phones, and the like.

Those of skill in the art can appreciate that a detailed discussion ofthe technology and processing required to implement noise reduction withthe use of two or more mics 504 is not needed to understand the aspectsof the embodiments. Nonetheless, the following is provided forcompleteness. Regardless of how far away the source of the audio signal,i.e., the voice command, one signal from a respective mic 504 will bestronger than the other. The two sound waves can be compared followingfiltering, digitization, and other processing. The non-voice signal, orthe one with the lower amplitude can be subtracted from the other,meaning the voice or audible command signal is now cleaner, with lessnoise.

According to aspects of the embodiment, the ASN can be part of a largercontrol system, such as control network 100. While the ASN can be partof control network 100, or the ASN can operate autonomously, referencefrom hereon in shall be made to only the ASN. As shown in FIG. 5,microphone 502 can incorporated into existing devices such as keypads500 and motion detectors, or can be stand-alone independent devices withcommunications capabilities such as IEEE 811.14 PAN transceiver 514.Accordingly, each device can have local voice recognition capabilities(i.e., through the use of a “mini” processor that is co-located with thestandalone microphone), of can be part of a centralized voicerecognition system wherein voice recognition processing occurs at aremote server (such as gateway 114 or controller 116) or a combinationof the two.

Attention is now directed towards FIG. 6, which illustrates thescientific principles upon which time-date stamping and amplitudedetermination are based. In FIG. 6, the source is shown as generating asound; in this case, a “light-off” command directed towards room 1. Thesound waves, as indicated, travel in the direction of arrow A, towardsfirst and second microphones 504 a 1 and 504 b 1 of keypad 106 a in room1, and third and fourth microphones 504 a 2 and 504 b 2 of keypad 106 bin room 2. The sound waves arrive at first and second microphones 504 a1,b 1 of keypad 106 a in room 1 at times T_(1′) and T_(1″),respectively, with amplitudes of A_(1′) and A_(1″), respectively,wherein first microphone 504 a 1 is a distance d_(1′) from the soundsource, and second microphone 504 b 1 is a distance d_(1″) from thesound source. Similarly, the sound waves arrive at third and fourthmicrophones 504 a 2,b 2 of keypad 106 b in room 2 at times T_(2′) andT_(2″), respectively, with amplitudes of A_(2′) and A_(2″),respectively, wherein third microphone 504 a 2 is a distance d_(2″) fromthe sound source, and fourth microphone 504 b 2 is a distance d_(2″)from the sound source. The calculations for determining the time in viewof the velocity of sound are known to those of skill in the art, andtherefore have been omitted herein in fulfillment of the dual purposesof clarity and brevity.

Accordingly, as the amplitude of the sound waves generally decreaseswith time and distance, the sound waves arriving at second microphone504 b 1 should be somewhat smaller in amplitude, and arrive at a latertime than those that arrive at first microphone 504 a 1. And, the soundwaves at third microphone 504 a 2 should be smaller and arrive at alater time than those of second microphone 504 b 1, and so on for thefourth, and other microphones, depending on their spatial location withrespect to the source of the sound. However, as those of skill in theart can appreciate, sometimes sound waves reflect off objects, causinglarger amplitudes at farther distances, or become attenuated for avariety of reasons that might be different from one location to theother, even if the locations within a setting are within relativelyshort distances (meters, or yards), depending on the construction of thehome or enterprise location. Thus, according to further aspects of theembodiments, amplitude or even time stamping might not be sufficientlydispositive in some cases in regard to the determination of which roomthe audible command is being directed towards, but in thosecircumstances they can be useful factors to take into account.

FIG. 7 illustrates a plan view of a floor of a house in which the systemand method for determining which controllable device an audible commandis directed to can be used according to aspects of the embodiments. Sucha setting as discussed above can be as realized in FIG. 7, wherein room1 is “Jordyn's Room” and room 2 is “Nolan's Room.” Someone has just leftroom 1, at position (1), carrying bags in each hand, and cannot hit thelight switch to turn off the lights on the way out. So, the person usesan audible command “Lights off,” as they pass position (2). However,both of microphone 504 a 1,b 1 (Jordyn's room, room 1) in keypad 106 a,and both of microphones 504 a 2,b 2 (Nolan's room, room 2) in keypad 106b, receive the audible command, and there can be confusion as to whichlights to turn off. There could be someone still in Nolan's room, and toturn off those lights could be dangerous, or at least inconvenient.Since the first pair of microphones 504 a 1,b 1 received the commandearlier (through comparison of the time-date stamp), the controllabledevice associated with the first pair of microphones 504 a 1,b 1 ofkeypad 106 a will be directed to respond to the audible command. Thisprocessing decision can also be made, or verified, by comparingamplitudes of the first and second digital signals—amplitude A₁ will begreater, albeit by a small amount, than amplitude A₂, and as suchwhichever processor processes the received digital signals, it willascertain that the command was directed to Jordyn's room, room 1,because the amplitude of the received signal is greater at Jordyn's roomthan at Nolan's room. According to aspects of the embodiments, the userwould prefer that the lights in Jordyn's room be turned off well beforethey get to position (3)—the bottom of the stairs.

According to further aspects of the embodiments, the ASN can furtherinclude proximity sensors as a further means for discerning the presenceor location of a user, which can assist in determining which room thecommand to the controllable device is directed towards. For example, inFIG. 7, there are shown a plurality of proximity sensors 702 a-d, oneeach for Jordyn's room, Nolan's room, Raegyn's room, and the bathroom,as well, as 702 e for the hallway. According to further aspects of theembodiments, proximity sensors can be the same or different as occupancysensors; that is, an occupancy sensor can be a passive detection device;motion, heat, among other types. Proximity sensors can be active—usingNFC, Bluetooth, Wi-Fi, or other low- or medium-power communicationsprotocols, that transmit signals to which a device, such as PED 104, canrespond to, thereby tracking movement and position of the user, withoutthe user's input. For example, in FIG. 7, the user, when at position(2), has left Jordyn's room and is now at the top of the stairs. Whileeach of proximity sensors 702 a-e can, most likely, detect the presenceof the user, each will have a different power level received signal fromPED 104 that the user is carrying with them. Of course, as those ofskill in the art can appreciate, this means that the user has to haveloaded onto their PED 104 a configuration application that contains aportion of the program dedicated to the particular low- or medium-powercommunications protocol being used by the ASN. Thus, a centralprocessor, such as central processor 116 or gateway 114 will receivedata from each of proximity sensors 702 and will be able to check thesignal strength level from each of proximity sensors 702 a-e. Theprocessor will therefore know that (a) the user has just left Jordyn'sroom, (b) is now headed down the stairs, and (c) that a command has justbeen issued by the user to turn off lights. A review of the light statusof each of the rooms on that floor will determine that the lights wereleft on in Jordyn's room, and a command can now be generated andtransmitted to turn them off.

While any one of the above proscribed processes can effectively turn offthe light in the room as intended by the audible command—speechrecognition algorithms, amplitude comparisons, time-date stampcomparisons, those of skill in the art can appreciate that additionalproblems can, from time to time, arise in the system and can potentiallybe the cause in erroneous operation. In some cases, any two of the threeprocesses can be combined, or all three can be used. In addition, asimmediately described below, additional processes can be implemented toconstruct an ASN that can operate substantially effectively,substantially all of the time.

As described above, according to aspects of the embodiments, one or moreprocessors can obtain received digital audible signals (e.g., DAS₁ andDAS₂), and can process each of DAS₁ and DAS₂ using speech recognitionalgorithms to determine the nature of the command, if any, contained inDAS₁ and DAS₂. If the command can be ascertained with a degree ofcertainty that meets or exceeds a predetermined degree of certainty(those of skill in the art can appreciate that currently availablespeech recognition algorithms can assign a value of certainty in regardto recognition of the speech of the received digital audio signals),then the controllable device to which the received digital audiblesignal is directed can be instructed to act on the command. Sometimes,however, such received digital audible signals are not recognizable bythe speech recognition algorithms. In these cases, additional processingcan be necessary to ascertain the device to which the received DAS isdirected. In this latter case, a time-date stamp can be applied to eachreceived DAS, and then those time-date stamps can be compared to make adetermination as to which DAS occurred first. The controllable deviceassociated with the microphone that received the first DAS can then bedirected to act on the command. In addition to comparing time-datestamps, the amplitude of the received DAS can also be compared; thecontrollable device associated with the microphone that received the DASwith the larger amplitude can be considered to be the one to which thecommand was directed. Amplitude comparisons, time-date stamp comparison,and speech recognition can be used independently of each other, or invarious combinations with each other. Other processes can also be used,as described below.

According to further aspects of the embodiments, to reduce falsepositives, the ASN can include speech recognition algorithm (SRA) thatrecognizes and distinguish audible commands from silence. The SRA canlearn, over time, the ambient noise levels of the room in which arespective microphone is located. According to an embodiment, theseambient levels become characterized as “silence” in the sense that theydo not convey useful command and control information, or can be actuallyvery low noise/sound level situations. According to further aspects ofthe embodiments, the SRA can then determine a state of silence (orabsence of a command), a state that a command has been issued, and thensilence again.

To reduce collocation errors, the SRA of the ASN recognizes zonecommands. For example, the SRA of the ASN can recognize commands such as“Master Bathroom Off” and “Guest Room On,” among others. In order toreduce errors in the acoustic sensory network, a user can speak suchcommands in a learning mode to that the SRA can learn to recognize thedifferent vocal traits of the user or users. According to furtheraspects of the embodiments, any number of users can input “practice”commands that can then be learned by the SRA. However, as those of skillin the art can appreciate, the SRA and acoustic sensory network is notnecessarily limited to such “learned” commands, nor does it even requiresuch learned commands in order to recognize zone commands. However, byknowing the different zones beforehand, the SRA and acoustic sensornetwork can increase its efficiency in recognizing and responding toreceived DASs.

According to still further aspects of the embodiments, to reduce falsepositives and collocation problems, occupancy indicators can be utilizedto determine location and deduce likely commands. For example, once anoccupancy sensor determines occupancy in a room and automaticallytriggers the lights to turn on, the acoustic sensor network can thendeduce that an “Off” command is likely to follow in that room.Similarly, other types of occupancy indicators can be combined, such asAV equipment operation, among others. Further, if a command is receivedby two or more co-located microphones, the occupancy sensors of each ofthe rooms can be checked, and if one still indicates an occupied roomwith the lights on, then it is likely the “Off” command was not directedtowards it. However, additional processing can be implemented that takesinto account time of day, day of the week, additional commands receivedwithin specified periods of time (e.g., correcting commands; a first“Off” command followed by a second “Off” command seconds later), amongother processes.

According to further aspects of the embodiments, additional processingsteps can be used to implement additional features. For example, toreduce privacy concerns, the ASN, at start-up, can temporarily employ aremote server to learn the operating environment of the keypad. Once anoise signature of the environment is deduced, the device may use alocal processor to filter out background noise and recognize commands.

According to further aspects of the embodiments, the ASN, which is partof control network 100 (or which can be a stand-alone network), canreduce background noise to make determination of commands via SRAs moreeffective. In order to reduce background noise, the ASN can request thatcertain noise producing devices be turned off in an area where a commandis likely to be heard. For example, after detecting lack of occupancy ina bedroom, the ASN can reduce the volume of any audio/video devices thatmay still be operating, and also reduce heating or air conditioning tothe room; this can lower the ambient background noise, and makedetection and determination of any commands easier.

According to further aspects of the embodiments, the ASN can reduce oreliminate collocation problems. As described above, collation problemsare those that related to two or more microphone devices 504 that arerelatively close to one another on two separate keypads 106 fordifferent rooms; see, e.g., keypads 106 a, 106 b, in FIG. 7 for Jordynand Nolan's room. One manner of reducing or substantially eliminatingcollocation issues, especially in systems that do not utilize zonecommands, is to check for occupancy in different rooms/areas. Afterdetecting occupancy in an area or room, the ASN can then disablemicrophones 504 known to be in adjacent zones (areas/rooms). Accordingto further aspects of the embodiments the ASN can also use occupancysensor data that shows no-occupancy as a means for reducing orsubstantially eliminating collocation issues. In this latter case,occupancy sensors from all adjoining rooms are checked against eachother; if a room fails to show occupancy, then commands for that roomthat room would turn off lights are ignored. As those of skill in theart can appreciate, there are numerous variations on how to useoccupancy sensor data by the ASN in processing commands to controlcontrollable devices such as lights.

According to still further aspects of the embodiments, the SRA of theASN can recognize characteristic sounds and deduce likely commands to bereceived in the vicinity. For example, a flushed toilet and runningwater are likely indicators that a bathroom microphone will receive a“lights off” command. A further example can be a garage door closing,among others.

According to still further aspects of the embodiments, the ASN can usevoice or speech recognition to identify speakers, learn preferences, andset defaults accordingly. For example, the SAR algorithm of the ASN canrecognize that certain individual users prefer certain temperature,humidity, AV, shade, and/or light settings, and set devices at thosesettings when detecting the user is present. When multiple users arepresent, the ASN can determine an optimal setting by taking into accounteach user's preference. According to still further aspects of theembodiments, when multiple users are present, the ASN can determine anoptimum setting by taking into account each user's preference.

Attention is now directed towards FIG. 8 that illustrates a flow chartof method 800 for determining which controllable device out of aplurality of controllable devices an audible command is directed towardsaccording to aspects of the embodiments.

As described herein, an encoding process is discussed in reference toFIG. 8 and method 800. The encoding process is not meant to limit theaspects of the embodiments, or to suggest that the aspects of theembodiments should be implemented following the encoding process. Thepurpose of the following encoding process is to facilitate theunderstanding of one or more aspects of the embodiments and to providethe reader with one or many possible implementations of the processeddiscussed herein. FIG. 8 illustrates a flowchart of various stepsperformed during the encoding process. The steps of FIG. 8 are notintended to completely describe the encoding process but only toillustrate some of the aspects discussed above. The encoding process canbe further embodied in one or more programs that reside in one or morememory locations of one or more devices, such as, for example, VR SoCcircuit 520, controller 116 and gateway 114, among other devices.However, in fulfillment of the dual purposes of clarity and brevity,discussion shall be made of method 800 as embodied in audible commandprocessing and determination (ACPD) program 222 (shown in FIG. 2) thatresides in memory 206 and can include one or more of AEC, DOA, and DABFprocessing, as well as SRAs.

Method 800 begins with optional method step 802. Each of the operationsof method steps 802-810 has been described in greater detail above inregard to FIGS. 1-7; therefore, in fulfillment of the dual purposes ofclarity and brevity, a detailed discussion of the same operations andsystem devices has been omitted from the discussion below of FIG. 8 andmethod 800. In method step 802, the ASN can be used to learn a user'sspeech characteristics in the manner as described above. In addition,method 800 can also acquire information regarding the zones or rooms ofa building, office, home, or enterprise location. The latter informationcan be used to assist in determining which zone or room an audiblecommand is directed towards as described in greater detail below. Method800 can use the zone/room information to determine the intended “target”of the command by matching control devices to the rooms and zones, andverifying the presence and operational status of the proximity andoccupancy sensors, and their locations with regard to each of the roomsand zones.

In method step 804, method 800 receives audio information in the form ofan analog signal at one or more microphones 504 at one or more keypads106 according to aspects of the embodiments. As described in greaterdetail above, the analog signals are digitized, time and date stamped,can be further processed to substantially eliminate or reduce noise(using AEC, DOA, DABF), and prepared for further processing. The audiosignal is now in the form of digital data, or packets, and can berepresented as AF_(n)(t), where n ranges from 1 to the total number ofkeypads 106 that report an audible signal. In decision step 806 ofmethod 800, each of the received plurality digitized audio signalsAF_(n)(t) is analyzed by a speech recognition algorithm (SRA) in ACPDprogram 222 in order to determine which keypad 106 and controllabledevices the audio command signal is directed towards. According to onenon-limiting aspect of the embodiments, for example, method 800 (andcertain modules of ACPD program 222) can be directed to the control oflighting devise; therefore, each command is therefore understandable inthe sense that is directed to turning lights on or off, or up or down inintensity, but the question is generally which room or zone is thecommand directed towards? As those of skill in the art can appreciate,however, the example of method 800 is not limited to lighting devicesonly, but can be used in controlling a plurality of different devices ashas been described herein. With modifications that have been describedherein method 800 can be used to control audio-video, HVAC, shading,security, and many other types of devices and/or systems, alone or incombination with each other. These can be embodied in one larger versionof ACPD program 222, or can be embodied in multiple modules of ACPDprogram 222 as the case may be.

If the output of the SRA is of a certainty that exceeds a predeterminedthreshold of certainty, then method 800 can direct the command to theappropriate device (“Yes” step from decision step 806). If the output ofthe SRA is such that the controllable device of the audio signal cannotbe discerned (“No” path from decision step 806), then method 800proceeds to decision step 808. In decision step 808, method 800 comparesthe date-time stamp and magnitude of each of the received AF_(n)(t)signals, presuming there is more than one (if there is only one, thenmethod 800 applies the command to the controllable device to which thecontrol device received the audible signal AF_(n)(t)). According toaspects of the embodiments, the magnitude of the earliest signal shouldbe greater than the magnitude of later arriving signals. Thus, by way ofexample, if there are two audible signals, AF₁(t₁) and AF₂(t₂), t₁should be less than t₂, and the magnitude of AF₁(t₁) should be greaterthan the magnitude of AF₂(t₂). If this is the case (“Yes” path fromdecision step 808), method 800 proceeds to step 807 and applies thecommand to the controllable devices associated with the first keypad106. The controllable devices of the control device that reports theearliest time-date stamp and greatest magnitude of AF_(n)(t) are thecontrollable devices to which the command contained in the receivedAF_(n)(t) signal will be directed towards. In this manner, method 800takes into account the fact that the speed of sound in a home or officeof enterprise location is essentially and substantially constant, andthus the control device that reports the earliest or greatest magnitudeaudible signal is the one that the command is directed to. If theseequalities do not hold true (“No” path from decision step 808), thenmethod 800 and ACPD program 222 are directed to decision step 810 forfurther processing and determinations according to aspects of theembodiments.

As those of skill in the art can no doubt now appreciate in view of thediscussion above, there are situations and cases where the proscribedprocess steps 802-808 of method 800 might not be enough to discern whichcontrollable device to direct the command to. Thus, in method step 810,additional parameters/processing/factors can be taken intoconsideration. As each of these has been described in greater detailabove, they will be only briefly discussed again at this point infulfillment of the dual purposes of clarity and brevity.

In method decision step 810, additional processing can be performed todetermine which room or zone or controllable device the received audiblesignal is directed towards. One, some, or all of the followingprocessing steps/parameters can be taken into account and/or performed,in any particular order, or none can and decision step 800 can terminatewith step 808, as described above.

According to still further aspects of the embodiments, a further stepthat can be taken in decision step 810 alone or in conjunction with oneor more of the other processing steps can be to make use of proximitysensors to determine which controllable device the audible signal(command) is directed to. That is, the system and method according toaspects of the embodiments, can use the knowledge of the presence, oflack of presence of a user as determined by one or more proximitysensors can be used to determine which controllable device the commandis directed to. This can be accomplished through the use of low/mediumpower communications protocols such as Bluetooth, NFC, Wi-Fi, amongothers. According to aspects of the embodiments, a transmitted signalinterrogates an electronic device such as PED 104; appropriate softwarelocated therein received such a transmission, and responds in kind tothe proximity sensor. A plurality of proximity sensors can send suchlocation interrogation transmissions. Upon processing all of theresponses from each proximity sensor, and determination can be made asto location based on the strength and/or time stamp of the signaltransmitted by PED 104 and received by each of the proximity sensors.

According to still further aspects of the embodiments, a further stepthat can be taken in method step 810 alone or in conjunction with one ormore of the other processing steps can be to make use of backgroundand/or ambient noise in a passive measure. That is, the system andmethod according to aspects of the embodiments can use microphones 504to periodically measure the background of ambient noise levels from timeto time and one or more processors, wherever located, can store suchreadings. These noise levels can then be subtracted from future readingsin order to facilitate the presence of commands. As those of skill inthe art can appreciate, while such a determination may not in and ofitself tell method 800 which controllable device the command is directedto, it can assist in helping to determine when a command has beenissued, and in conjunction with other processing steps and/or parameterdeterminations can be used to determine the nature of the command (e.g.,which controllable device in which zone/room the command is directedtowards).

According to still further aspects of the embodiments, a further stepthat can be taken in method step 810 alone or in conjunction with one ormore of the other processing steps can be to make active use of theoccupancy sensor data. That is, the system and method according toaspects of the embodiments can determine the presence or not of a personor persons in a room, and further determine when such person or personsleave the room. Then it can be expected that any ensuing command couldbe directed towards that room or zone that was just vacated.

According to still further aspects of the embodiments, a further stepthat can be taken in method step 810 alone or in conjunction with one ormore of the other processing steps can be to make active use of theoutput of the occupancy and proximity sensors. That is, the system andmethod according to aspects of the embodiments can turn off or reduce involume any “noise” producing devices when occupancy and/or proximitysensors suggest that a room or zone is unoccupied. By reducing theambient noise level—in this case, noise referring to any audible soundthat is not a command (e.g., music, video audio)—the commands, or anycommands that might be issued, will become easier to discern, andrespond to.

According to still further aspects of the embodiments, a further stepthat can be taken in method step 810, alone or in conjunction with oneor more of the other processing steps, can be to make active use of theoutput of the occupancy and proximity sensors in another, differentmanner than that just described. That is, the system and methodaccording to aspects of the embodiments can turn off or disable one ormore microphones 504 in one or more rooms or zones when occupancy and/orproximity sensors suggest that a room or zone is unoccupied. Byeliminating one or more outputs from microphones 504 in which a commandis not expected, the system and method according to aspects of theembodiments will be able to detect with greater accuracy thecontrollable device that the command is being directed towards.

According to still further aspects of the embodiments, a further stepthat can be taken in method step 810 alone or in conjunction with one ormore of the other processing steps can be to make active use ofbackground and/or ambient noises; that is, when certain background orambient noises occur, the system and method according to aspects of theembodiments can predict future operations based on those noises. Suchpredictive behavior can be learned over time. By way of non-limitingexamples, when a user closes a garage door, for example, a command toturn off the lights in the garage can be expected. Similarly, when atoilet flushes, or running water is turned off in the bathroom sink, thecommand to turn off the lights in the bathroom can be expected.

Once one, some, or all of the above additional processing forascertaining the correct room to which the received audible command isdirected towards are performed, method 800 (ACPD program 222) candetermine whether, within a certain predefined degree of certainty as towhich room the command is directed. If method 800 can make thedetermination (“Yes” path from decision step 810), method 800 proceedsto method step 807 wherein the command is applied to the specifiedcontrollable device of the specified room or controlled zone. If,however, after all of the processing of method 800 as embodied as ACPDprogram 222, a determination still cannot be made, then no response isprovided, and the command is ignored. The user or users can be informedof this by some type of audible, visual, or haptic feedback, or anycombination thereof (including all of the feedback methods).

FIG. 9 illustrates processing and memory components/circuitry of one ormore of the personal electronic device 104 of FIG. 4, gateway device 114of FIG. 3, controller 116 of FIG. 2, and any other devices that uses oneor more processors as described herein that uses software and/orapplications to perform various functions and actions as describedherein according to aspects of the embodiments.

FIG. 9 illustrates processing and memory components/circuitry (generallyreferred to as a “computer” or PC) of one or more of the personalelectronic device 104 of FIG. 4, gateway device 114 of FIG. 3,controller 116 of FIG. 2, and any other devices that uses one or moreprocessors as described herein that uses or runs or implements softwareand/or applications, such as the configuration application, projectfiles, or control applications, to perform various functions and actionsas described herein according to aspects of the embodiments, suitablefor use to implement method 800 for determining which controllabledevice an audible command is directed towards according to anembodiment.

PC 900 comprises, among other items, integrated display/touch-screen 902(though not used in every application of PC 900), internal data/commandbus (bus) 904, processor board/PC internal memory (internal memory) 932,and one or more processors 908 with processor internal memory 906 (whichcan be typically ROM and/or RAM). Those of ordinary skill in the art canappreciate that in modern PC systems, parallel processing is becomingincreasingly prevalent, and whereas a single processor would have beenused in the past to implement many or at least several functions, it ismore common currently to have a single dedicated processor for certainfunctions (e.g., digital signal processors) and therefore could beseveral processors, acting in serial and/or parallel, as required by thespecific application. PC 900 further comprises multiple input/outputports, such as universal serial bus ports 910, Ethernet ports 911, andvideo graphics array (VGA) ports/high definition multimedia interface(HDMI) ports 922, among other types. Further, PC 900 includes externallyaccessible drives such as compact disk (CD)/digital versatile disk (DVD)read/write (RW) (CD/DVD/RW) drive 912, and floppy diskette drive 914(though less used currently, many PCs still include this device).

Internal memory 932 itself can comprise HDD 916 (these can includeconventional magnetic storage media, but, as is becoming increasinglymore prevalent, can include flash drive memory 934, among other types),ROM 918 (these can include electrically erasable (EE) programmable ROM(EEPROMs), ultra-violet erasable PROMs (UVPROMs), among other types),and RAM 920. Usable with USB port 910 is flash drive memory 934, andusable with CD/DVD/RW drive 912 are CD/DVD disks 936 (which can be bothread and write-able). Usable with floppy diskette drive 914 are floppydiskettes 938. External memory storage 924 can be used to store data andprograms external to box 901 of PC 900, and can itself comprise anotherhard disk drive 916 a, flash drive memory 934, among other types ofmemory storage. External memory storage 924 is connectable to PC 900 viaUSB cable 956. Each of the memory storage devices, or the memory storagemedia (906, 916, 918, 920, 924, 934, 936, and 938, among others), cancontain parts or components, or in its entirety, executable softwareprogramming code or application (application, or “App”), such as ACPDprogram 222, which can implement part or all of the portions of method800 described herein.

Bus 904 provides a data/command pathway for items such as: the transferand storage of data/commands between processor 908, integrated display902, USB port 910, Ethernet port 911, VGA/HDMI port 922, CD/DVD/RW drive912, floppy diskette drive 914, and internal memory 932. Through bus904, data can be accessed that is stored in internal memory 932.Processor 908 can send information for visual display to either or bothof integrated and external displays, and the user can send commands tosystem operating programs/software/Apps 940 that might reside inprocessor internal memory 906 of processor 908, or any of the othermemory devices (936, 938, 916, 918, and 920).

PC 900, and either processor internal memory 906 or internal memory 932,can be used to implement method 800 for determining which controllabledevice an audible command is directed towards according to anembodiment. Hardware, firmware, software or a combination thereof can beused to perform the various steps and operations described herein.According to an embodiment, App 940 for carrying out the above discussedsteps can be stored and distributed on multi-media storage devices suchas devices 916, 918, 920, 934, 936 and/or 938 (described above) or otherform of media capable of portably storing information. Storage media934, 936 and/or 938 can be inserted into, and read by devices such asUSB port 910, CD/DVD/RW drive 912, and disk drives 914, respectively.

As also will be appreciated by one skilled in the art, the variousfunctional aspects of the embodiments may be embodied in a wirelesscommunication device, a telecommunication network, or as a method or ina computer program product. Accordingly, the embodiments may take theform of an entirely hardware embodiment or an embodiment combininghardware and software aspects. Further, the embodiments may take theform of a computer program product stored on a computer-readable storagemedium having computer-readable instructions embodied in the medium. Anysuitable computer-readable medium may be utilized, including hard disks,CD-ROMs, DVDs, optical storage devices, or magnetic storage devices sucha floppy disk or magnetic tape. Other non-limiting examples ofcomputer-readable media include flash-type memories or other known typesof memories.

Further, those of ordinary skill in the art in the field of theembodiments can appreciate that such functionality can be designed intovarious types of circuitry, including, but not limited to fieldprogrammable gate array structures (FPGAs), ASICs, microprocessor basedsystems, among other types. A detailed discussion of the various typesof physical circuit implementations does not substantively aid in anunderstanding of the embodiments, and as such has been omitted for thedual purposes of brevity and clarity. However, as well known to those ofordinary skill in the art, the systems and methods discussed herein canbe implemented as discussed, and can further include programmabledevices.

Such programmable devices and/or other types of circuitry as previouslydiscussed can include a processing unit, a system memory, and a systembus that couples various system components including the system memoryto the processing unit. The system bus can be any of several types ofbus structures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures.Furthermore, various types of computer readable media can be used tostore programmable instructions. Computer readable media can be anyavailable media that can be accessed by the processing unit. By way ofexample, and not limitation, computer readable media can comprisecomputer storage media and communication media. Computer storage mediaincludes volatile and nonvolatile as well as removable and non-removablemedia implemented in any method or technology for storage of informationsuch as computer readable instructions, data structures, program modulesor other data. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, DVD,or other optical disk storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canbe accessed by the processing unit. Communication media can embodycomputer readable instructions, data structures, program modules orother data in a modulated data signal such as a carrier wave or othertransport mechanism and can include any suitable information deliverymedia.

The system memory can include computer storage media in the form ofvolatile and/or nonvolatile memory such as ROM and/or RAM. A basicinput/output system (BIOS), containing the basic routines that help totransfer information between elements connected to and between theprocessor, such as during start-up, can be stored in memory. The memorycan also contain data and/or program modules that are immediatelyaccessible to and/or presently being operated on by the processing unit.By way of non-limiting example, the memory can also include an operatingsystem, application programs, other program modules, and program data.

The processor can also include other removable/non-removable andvolatile/nonvolatile computer storage media. For example, the processorcan access a hard disk drive that reads-from or writes-to non-removable,nonvolatile magnetic media, a magnetic disk drive that reads from orwrites to a removable, nonvolatile magnetic disk, and/or an optical diskdrive that reads from or writes to a removable, nonvolatile opticaldisk, such as a CD-ROM or other optical media. Otherremovable/non-removable, volatile/nonvolatile computer storage mediathat can be used in the operating environment include, but are notlimited to, magnetic tape cassettes, flash memory cards, digitalversatile disks, digital video tape, solid state RAM, solid state ROMand the like. A hard disk drive can be connected to the system busthrough a non-removable memory interface such as an interface, and amagnetic disk drive or optical disk drive can be connected to the systembus by a removable memory interface, such as an interface.

Aspects of the embodiments provide for the use of unrelated multimediadevices while microphones are listening for voice commands. Thistechnology also allows for intercoms and phone calls with improvedfidelity in the presence of multimedia devices playing in the vicinity.

FIG. 11 illustrates conventional voice recognition system (CVRS) 1100.CVRS 100 comprises CVRS circuitry (CVRSC) 1102, video distributionsystem (VDS) 1150, internet 1126, and voice recognition server (VRSv)1110, and interfaces with person 1122. Person 1122 speaks and generatesunknown voice audio 1124, which CVRS attempts to decipher, and act upon,in the presence of unknown multi-media audio (MMA_(U)) 1120 broadcast byvideo distribution system (VDS) 1150, as well as audio broadcast bymulti-media audio system (MMAS) 1138, which also can be referred to asMMA_(U) 1120. VDS 1150 comprises high-definition multimedia interface(HDMI) transceiver 1128, display 1134, and audio sound bar 1132. CVRSC1102 comprises VRS processor 1104, network connection 1106, one or morespeakers 1114, and one or more microphones (mic) 1116. It is fairly wellknown by many people at this time how CVRSC 1102 operates. Eachdifferent type of CVRSC 1102 substantially continuously samples theoutput of an electrical transducer—a microphone—for the presence of aspecific audio signal that is converted to an electrical signal. In manycases, the specific electrical signal represents electrically a knownspoken word, such as “Siri®” or “Alexis®.” These words will have acertain, known frequency response, which CVRSC 1102 can recognize,following digitization (such well known circuitry and details have beenomitted from FIG. 11, in fulfillment of the dual purposes of clarity andbrevity). When it is determined that this known command word has beenspoken, CVRSC 1102 and VRSv 1110, using one or more voice recognitionprograms or algorithms, acquires and deciphers the spoken words thatfollow to determine what they are, discern their meaning, and respond torequests. Operation of CVRSC 1102, VRSv 1110, and its voice recognitionalgorithms are discussed in greater detail below. At this point in time,many people are aware of how such systems operate, even if theythemselves do not own and operate the technology.

MMAS 1138 transmits electrical signals over speaker cable 1108 (orwirelessly, but such circuitry has not been included in FIG. 11) thatare broadcast by transducers (speakers) 1114 b, which can be located inthe ceiling and/or walls or other locations within a room or building,or even outside the building. Such configurations are well known tothose of skill in the art. The audio signals broadcast by speakers1114,1132 can be referred to as MMA_(U) 1120, and can consist of voice,music, and/or other types of sounds. The audio output by speakers 1114b, as well as other devices described below, is referred to as “unknown”because it is not known beforehand by CVRSC 1102 prior to beingbroadcast, and thus received by microphone 1116, and input to VRSprocessor 1104 and VRSv 1110. Consequently, because it is unknown audio,the voice recognition software/algorithms have greater difficultycompensating for it in determination the received voice audio. This isdiscussed in greater detail below.

HDMI transceiver 1128 receives video and audio signals either from alocal or remotely located video source, such as a DVD player (notshown), or some other similar type of device, or via network/Internet1126. Network 1126 can be virtually any type of network, such as a PAN,LAN, wide area network (WAN), the “cloud,” micro networks, among others,as well as combinations thereof. The output of HDMI transceiver 1128 areelectrical signals transmitted over HDMI cable 1130 (or wirelessly, notshown) and contain video and MMA_(U) 1120 signals, the former of whichis displayed by video display 1134, and the latter is broadcast by audiosound bar 1132. Audio sound bar 1132 typically contains severalspeakers, generally a mix of low frequency range, mid-frequency range,and high-frequency range speakers (e.g., 20-120 Hz, 100 Hz-10 kHz, and8-20 kHz, respectively). The output of audio sound bar 1132 is MMA_(U)1120, i.e., another unknown type of audio.

In addition to the unknown audio broadcast by audio sound bar 1132, andthe unknown audio broadcast by speakers 1114 b, there is also shown inFIG. 11 one or more person(s) 1122 whom generate voice audio 1124, whichis also an unknown type of audio signal. Thus, in FIG. 11, there are atleast three sources of unknown audio: external MMAS 1138, HDMItransceiver 1128, and person 1122 (for each source-type, there can beone or more such sources). As those of skill in the art can appreciate,other types of unknown audio includes, but is not limited to,environmental sounds (wind, rain, sleet, hail, among other types),machine sounds (fans, motors, blowers. compressors, among other types),and background miscellaneous noise (other people talking, doors openingand closing, and sources of sound that are the same, or of similarnature.

As briefly discussed above, CVRSC 1102 comprises VRS processor 1104, mic1116, and speaker 1114. Processor 1104 transmits and receives audio andvoice responses over network connection 1106, which establishes aconnection to internet 1126 in a known, conventional manner. In most, ifnot all CVRSCs 1102, voice recognition is generally accomplished at botha remotely located server such as VRSv 1110, and in processor 1104, asshown in FIG. 1. Processor 1104 detects the presence of a keyword (suchas “computer”) prior to transmitting audio files to VRSv 1110. VRSv1110, accessible to CVRSC 1102 via internet/network 1126, is adapted tostore in memory one or more voice recognition algorithms. Digitizedsignals, typically transmitted using IP packet based technology, aretransmitted to VRSv 1110 where they are received and processed by thealgorithms, and audio responses are sent back to CVRS 1102 forbroadcasting by CVRS 1102 via one or more speakers 1114 a. As those ofskill in the art can appreciate, this means that each CVRSC 1102 willhave its own unique internet protocol address. As those of skill in theart can further appreciate, this description is greatly simplified, butexemplifies the basic operation of CVRSC 1102. As those of skill in theart can still further appreciate, any and all communications can be viawired or wireless means, or any combination thereof, and can include oneor more of cellular, satellite, NFC, BlueTooth, and WiFi (IEEE 802.11(any and all variations, thereof)), as well as other wireless protocols.

Speaker 1114 a outputs known multi-media audio (MMA_(K)) 1118 that ithas received via internet/network interface 1106 that forms part ofCVRSC 1102. Because CVRSC 1102 has knowledge of MMA_(K) 1118 before itis broadcast, and received by its own mic 1116, it can decipher unknownvoice audio 1124 in the presence of MMA_(K) 1118, if no other, orsubstantially no other, unknown multi-media audio MMA_(u) 1120 ispresent; however, those of skill in the art can appreciate that thatscenario is rarely the case; that is, it is generally the case thatthere will always be background noise, or music, or other multi-mediaaudio being broadcast from an unknown source. Such unknown sources canbe external audio devices, such as external multi-media audio system1138, and the multi-media audio output by HDMI transceiver 1128. It isthe presence of unknown audio that makes it difficult to use CVRSC 1102many times, because CVRSC 1102 and VRSv 1110 has increased difficulty indiscerning the spoken commands in the presence of what is essentiallybackground noise.

FIG. 12 illustrates a block diagram view of external audio compensatedvoice recognition system (EAC-VRS) 1200 according to aspects of theembodiments, wherein previously unknown multi-media audio MMA_(U) 1120,which can be broadcast by VDS 1150, and/or audio from VRSv 1110, becomesknown multi-media audio (MMA_(K) 1118), and EAC-VRS 1202, according toaspects of the embodiments, can take into account the known multi-mediaaudio MMA_(K) 1118 to enhance the recognition of voice audio 1124generated by person 1122 speaking, according to aspects of theembodiments.

EAC-VRS 1200 is adapted to interface with many of the same components asin CVRS 1102, and as such their operation need not be explained again,in fulfillment of the dual purposes of clarity and brevity. EAC-VRS 1200further comprises EAC-VRS circuitry (EAC-VRSC) 1202, remotely locatednetwork microphone devices (NMD) 1212 1116, HDMI audio extractor device1206, and audio amplifier 1210 according to aspects of the embodiments.Prior to discussing operation of EAC-VRS 1200, each of the newcomponents of EAC-VRS 1200 will be described.

In FIG. 11, VDS 1150 was shown and described, and in FIG. 12, many ofthe same components exists in VDS 1220 as in VDS 1150. There are,however, some differences. For example VDS 1220 as shown in FIG. 12further comprises HDMI audio extractor 1206, which is shown in greaterdetail in FIG. 15. Referring now to FIG. 15, HDMI audio extractor 1206comprises HDMI transceiver 1502, which receives HDMI video and audiofrom HDMI transceiver 1128. Ostensibly, these two transceivers can bethe same device, or different, in that HDMI transceiver 1128 is aninitial or primary transceiver of HDMI video from either a stand-alonesource, or the internet 1126. In addition, it is possible that HDMItransceiver 1502 may not, in some circumstances, be necessary to receivethe HDMI video from HDMI transceiver 1128 according to aspects of theembodiments. In this case, HDMI video would be received directly by HDMIvideo and audio delay 1504. Following HDMI transceiver, the receivedcombined video and audio signal is sent to HDMI audio extractor (audioextractor) 1506 a, and HDMI video and audio delay (delay) 1504.

Audio extractor 1506 a receives the un-delayed combined video and audiosignal and, according to aspects of the embodiments, extracts the audiosignal and sends it to audio transceiver 1208 a. The audio signal cannow be referred to as MMA_(K) 1118, and transmits it (as transmittedaudio signal 1218) to NMD 1212, which has a corresponding audiotransceiver 1208. Audio transceiver 1208 a can transmit transmittedaudio signal 1218 via wired or wireless means to NMD 1212. In contrastto the transmitted signal from audio transmitter 1208 b, discussedbelow, the audio signal transmitted by audio transmitter 1208 a is notdelayed. The purpose of delaying the audio signal is discussed ingreater detail below.

As shown in FIG. 15, the combined HDMI video and audio signal receivedfrom transceiver 1502 is also sent to delay 1504. Delay 1504, which isprogrammable, can delay the combined signal by a pre-determined,programmable amount of time. Delay 1504 can be programmed to delay thecombined video and audio signal from about 1 millisecond to about 10milliseconds. According to further aspects of the embodiments, differentranges of delay are also possible. Once delayed, the combined video andaudio delayed signal is output to video display 1134 and audio sound bar1132, as well as to HDMI audio extractor 1506 b. HDMI audio extractor1506 a extracts the audio from the delayed signal, and sends it to audiotransceiver 1208 b. Audio transceiver 1208 b receives the now-knowndelayed audio signal (referred to as MMA_(K) 1118′) and transmits it (astransmitted audio signal 1218), in either or both of a wired orwirelessly manner, to audio amplifier 1210, if one exists in EAC-VRS1200 according to aspects of the embodiments.

Referring back to FIG. 12, there is also shown audio amplifier 1210.Audio amplifier 1210 can receive as an input audio from MMAS 1138through audio delay and extraction device 1222, or can receive delayedaudio from HDMI audio extractor device 1206 as transmitted audio signal1218 by audio transmitted 1208 b. In this latter case, transmitted audiosignal 1218 contains delayed known audio, MMA_(K) 1118′. The receivedaudio is processed and amplified and sent to speakers 1114 b, which canbe located in one or more of ceilings, wall, or exterior portions, andbroadcast.

According to further aspects of the embodiments, audio delay andextraction device 1222, a detailed view of which is shown in FIG. 22,delays the unknown audio output from MMAS 1138. MMAS 1138 can be atuner, reel-to-reel tape machine, compact disk player, conventionalstereo system, a series of stereo components, turntable, network musicstreaming device, among other types of devices. The output of MMAS 1138is processed by audio delay and extraction device 1222. Referring now toFIG. 22, it can be seen that un-delayed audio MMA_(K) 1118 from MMAS1138 is received by transceiver-digitizer 2202, which can digitize theoutput of analog devices, such as a tuner, if needed. The output oftransceiver-digitizer 2202 is sent to delay 2204, which delays thedigitized/digital audio signal, MMA_(K) 1118′, by a known, predeterminedamount, and is sent to audio amplifier 1210; the other output oftransceiver-digitizer 2202 is sent to audio transceiver 1208 in audioextraction and delay device 1222, which transmits, either in a wired orwireless manner, un-delayed audio output signal MMA_(K) 1118 from MMAS1138 to NMD 1212, where it is sent to AEC 1304 and processed, asdescribed below.

Also shown in FIG. 12 is EAC-VRS NW processor 1204, a detailed blockdiagram of which is shown in FIG. 19. Referring now to FIG. 19, NWprocessor 1204 is shown and comprises microprocessor 1902, bus 1904, anddelays 1504 a,b, among other components not shown. As those of skill inthe art can appreciate, many components of a processor have beeneliminated from FIG. 19, as both beyond the scope of this discussion andnot needed to understand the aspects of the embodiments. Therefore, infulfillment of the dual purposes of clarity and brevity, illustrationand discussion of the same have been eliminated from herein. Audio data,regardless of its ultimate origin, is received from interface 1106 in1202, and placed on bus 1904, through command and control ofmicroprocessor 1902. Audio data destined to be sent to audio transceiver1208, and then to amplifier 1210, is put through programmable delay 1504b, and then to audio transceiver 1208. The audio signal is then referredto as MMA_(K) 1118′ (known audio, delayed). This can be music audiodata, but can also be audio data generated in VRSv 1110, according toaspects of the embodiments. Audio data destined to be sent to speaker1114, is put through programmable delay 1504 a, and then to speaker 1114a. The audio signal is then referred to as MMA_(K) 1118′ (known audio,delayed). This is typically audio data that originates from VRSv 1110.Audio data destined to be sent to NMD 1212, is transmitted directlythereto, and referred to as MMA_(K) 1118; in this case, this is anyaudio data received by processor 1204 according to aspects of theembodiments. The audio signal is then referred to as MMA_(K) 1118 (knownaudio, un-delayed).

Attention is now directed back to FIG. 12. According to aspects of theembodiments, by tying in what was previously unknown audio, MMA_(U)1120, from external sources, into EAC-VRSC 1202, MMA_(U) 1120 becomesknown audio, MMA_(K) 1118, and the voice recognition algorithms, whichneed either no or no-substantive modifications, can better discern voiceaudio 1124, as that now becomes virtually the only significant source ofunknown audio in the environment in which EAC-VRSC 1202 operates. Thatis, being able to subtract or eliminate more sources of audio as beingnot relevant audio makes the voice recognition software and algorithmsperform more effectively and efficiently, and therefor able to morereadily discern spoken audio commands 1124. EAC-VRSC 1202 and the voicerecognition algorithms can then better able determine what the voicecommands are, and then react to them more precisely.

According to aspects of the embodiments, and as shown in FIGS. 12 and15, unknown audio, MMA_(U) 1120, can be obtained by EAC-VRS NW processor1204 located in EAC-VRSC 1202 from extracted audio transceiver 1208located in HDMI audio extractor device 1206. In the case of sendingextracted audio to EAC-VRS NW processor 1204, HDMI extractor device 1206sends both delayed audio MMA_(K) 1118′ and un-delayed MMA_(K) 1118.EAC-VRS NW processor 1204 retrieves the audio, and broadcasts thedelayed version (MMA_(K) 1118′) through speaker 1114 a, and forwards theun-delayed version (MMAK 1118) to NMD 1212 through microphone interface1214 for use in acoustic echo cancellation processes, described ingreater detail below, in which the known audio is subtracted from thereceived outputs of mics 1306, as shown in FIGS. 13 and 14. Those ofskill in the art can appreciate that this description of operation ofthe AEC is greatly oversimplified.

As those of skill in the art can appreciate, the response by VRSv 1110(i.e., the response to a spoken query picked up by NMD 1212, is“unknown” audio until it is processed by EAC-VRS NW processor 1204, aswell as other audio received by network connection 1106 through network1126 (e.g., music from one or more streaming services, or music playedby MMAS 1138, among other sources and types of audio) according toaspects of the embodiments. Thus, the input to network connection 1106is shown to contain MMA_(U) 1120, and it becomes known after beingreceived and processed by EAC-VRS NW processor 1204.

In regard to HDMI video (and its accompanying audio) HDMI audioextractor 1206 processes the received video signals and audio signals,and in doing so, what was previously unknown audio becomes known audioMMA_(K) 1118. No physical or electrical or any other type oftransformation takes place other than insertion of a delay, as discussedabove in regard to FIG. 15. Once HDMI audio extractor obtains the audiosignals, and inserts a suitable delay, the otherwise unchanged audio (orcombined audio-video) signals are re-transmitted to their destinations.Delayed audio and video is sent to audio sound bar 1132 and videodisplay 1134. Delayed audio can also be sent to audio amplifier 1210 viaaudio transceiver 1208 in audio extractor 1206, and broadcast byspeakers 1114 b, as shown in FIG. 12.

According to further aspects of the embodiments, an external amplifiercan be used to power remotely located speakers that are not directlyconnected to the audio source, or EAC-VRSC 1202. These external speakers(not shown) can be installed in ceilings, walls, or even outside, andpowered by the external amplifiers (also not shown). The audio signalssent to the external speakers can be transmitted by audio transceiver1208 located in either audio extractor 1206, or in amplifier 1210

As those of skill in the art can appreciate, most microphone arrays usedfor voice recognition are located in a device that has speakers—e.g.,VRS 1102, as shown in FIG. 11. Some examples include a table top mediaspeaker like Amazon Echo®. Further, there are televisions and sound barslocated near televisions that contain mic arrays. These locations arenot ideal because there is a poor echo return loss (ERL) in thisconfiguration. ERL is the level of the sound coming from the speakersheard by the microphones compared to the level of the voice that themicrophones are intended to pick up. Poor ERL often results in poor echocancellation. To improve the ERL, it is desirable to locate themicrophones as far as possible from the speakers and closer to thelocation where the person speaking is located. Better locations of themicrophones include table top, ceiling mount and wall mount locations,among others.

Attention is now directed towards FIGS. 13 and 14. According to aspectsof the embodiments, NMDs 1212 will not only be remotely located fromspeakers 1114, but will have additional features that facilitate voicepickup. Because NMD 1212 is remotely located, the circuitry locatedwithin NMD can be powered by one or more of a hard-wire connection tohouse power, one or more batteries (which can also be rechargeable), anda PoE device. None of these power devices have been included shown inFIG. 13 or 14 in fulfillment of the dual purposes of clarity andbrevity, as those of skill in the art can appreciate how each of thesedevices operate. NMDs 1212 can each include a plurality of MEMsmicrophones 1306, as shown in FIGS. 13 and 14, which can be used in alinear (FIG. 13) or 2-dimensional configuration (FIG. 14) to enabledirection of arrival and adaptive beamforming.

As shown in FIG. 13, each of MEMs mic 1306 is followed by its own AECcircuit 1304; those of skill in the art can appreciate that while MEMsmics 1306 are generally stand-alone, individual devices, AEC circuit1304 can be configured in a package that includes eight or sixteen oreven more such circuits, each accepting at least two inputs: the firstbeing the output of MEMs mic 1306, and the other being the un-delayedknown audio signal, MMA_(K) 1118. In AEC circuit 1304, the known,un-delayed audio is subtracted from each of the outputs of the MEMs mics1306. MEM mics 1306 detect acoustical energy, which consists of thedelayed known audio, MMA_(K) 1118′ and unknown voice audio 1124, andconvert the same to electrical signals, in a manner well known to thoseof skill in the art.

AEC processing then occurs. According to aspects of the embodiments,once the electrical signals representing the voice signals are receivedby AEC circuits 1304, the analog electrical signals are digitized andtime-tagged, and processing ensues (in some instances, according tofurther aspects of the embodiments, each MEM mic 1306 output can belocation tagged, and converted to a digital signal as well). Accordingto further aspects of the embodiments, AEC circuit 1304 can also bebuilt into the microphone assemblies themselves; however, for thepurposes of this discussion, hereon in after, reference shall be made toreference signal MMA_(K) 1118 being received and processed in AECcircuit 1304, as shown in FIG. 13, although those of skill in the artcan now appreciate that such processing can also occur in mics 1116. Asshown in FIG. 13, AEC occurs prior to dynamic beamforming in directiondetection and beamforming (DDB) circuit 1302. Dynamic beamformingprocesses are well known to those of skill in the art, and therefore, infulfillment of the dual purposes of clarity and brevity, a detaileddiscussion thereof will not be repeated herein.

According to aspects of the embodiments, it is in AEC circuit 1304 thatpreviously unknown audio, MMA_(U) 1120, which is now known audio,MMA_(K) 1118 is used to isolate unknown voice audio 1124 from theoutputs of MEMS mic 1306. That is, MMA_(K) 1118 is used as a referencesignal in AEC circuits 1304 to help create a filter needed to eliminatethe sound of the unknown multimedia. According to aspects of theembodiments, if EAC-VRSC 1202 is being used in an intercom or phonemode, then MMA_(K) 1118 can also contain the far end voice signal sothat it is cancelled so that a full duplex call can be made. Accordingto further aspects of the embodiments, a mix of multimedia (e.g.,MMA_(K) 1118) and far end signal (e.g., voice) can be used as areference signal in AEC circuit 1304 so a conversation can be carried onwhile playing music. Sources of MMA_(U) 1120 can include one or more ofaudio transmitted through network 1126, and HDMI video and audio (viaVDS 1220).

According to still further aspects of the embodiments, the reference tothe microphone device, or AEC circuit 1304 (e.g., MMA_(K) 1118), is tobe received at each AEC circuit 1306 such that it has a substantiallyconstant latency between them all, meaning, the audio signal is receivedat each AEC circuit 1306 a-n at substantially the same time. In order tofacilitate a substantially equal arrival time at each AEC circuit 1306,a delay can be added to each path following receipt by EAC-VRS processor1204 (not shown). According to further aspects of the embodiments, AECcircuit 1304 can also compensate for relatively minor differences inarrival times between the acoustic signals that are converted by MEMsmic 1306. In addition, and according to further aspects of theembodiments, known audio MMA_(K) 1118 reaches AEC circuit 1304 prior towhen delayed known audio MMA_(K) 1118′ reaches MEMs mic 1302. Thisprinciple is shown in FIGS. 16 and 17.

FIG. 16 illustrates a timing diagram of delayed known multi-media audio,un-delayed multi-media audio, and unknown audio from a speaker and/orother sources according to aspects of the embodiments, and FIG. 17illustrates a simplified block diagram illustrating the principle ofdetermining a time delay between the broadcasting of known audio fromone or more speakers and receipt by a microphone associated with anacoustic echo cancellation device wherein knowledge of the delayincreases the efficacy of echo cancellation and other audio systemprocesses according to aspects of the embodiments.

FIG. 16 illustrates a timing diagram between delayed known audio MMA_(K)1118′, un-delayed known audio MMA_(K) 1118, and unknown audio 1124 inEAC-VRS 1200 according to aspects of the embodiments. According toaspects of the embodiments, insertion of a delay into the audio streamsuch that known audio is received at each of mics 1116/AEC circuit 1306before receipt of the same audio broadcast by speakers 1114 along withthe unknown voice audio 1124 provides for increased efficacy of echocancellation and other audio system processes. According to aspects ofthe embodiments, EAC VRS 1202 can incorporate delay D1 (shown in FIG. 16as Δt) between the source of each audio signal and speaker 1114 in orderto ensure that un-delayed audio MMA_(K) 1118 is received by AEC circuit1306 before the same delayed audio MMA_(K) 1118′ is received by AECdevice 1306.

Referring now to FIG. 16, the known audio MMA_(K) 1118 is delayed by Δt,which is the length of time between lines A and B. That is, knownbroadcast audio MMA_(K) 1118 is delayed by Δt, through audio delay 1504,located in both HDMI audio extractor device 1206, and in EAC-VRS NWprocessor 1204, and is processed concurrently with unknown voice audio1124 at the time shown by line B. For simplicity, the unknown audio isshown in the form of a square wave, and each pulse is numbered. As thoseof skill in the art can further appreciate, the delay is substantiallycontinuous, but in the diagram, it is shown as occurring at a singlepoint in time. As can be seen, at line A, the difference between the twosignals is two pulses. The delay processing “begins” at line A, whenpulse 4 appears in the un-delayed audio signal. In actuality, unknownvoice audio 1124 and delayed MMA_(K) 1118′, are combined. Because theun-delayed version arrives first at AEC circuit 1306, it has theopportunity (in this case, during the two pulse widths) to process thedelayed audio before pulse 4 combines with MMA_(U) 1124 and appears atthe input of AEC 1306, which occurs at line B. This is shown by line C;Since AEC circuit 1306 has knowledge of MMAK 1118′, it can subtract itfrom the combined signal of MMAU 1124 and delayed MMAK 1118′. Otherprocessing can occur as well.

In order to implement the delay, delay 1504, as shown in FIG. 15, can beinserted in extractor device 1206 and processor 1204, though this neednot necessarily be the case. That is, delay 1504 can also be astand-alone device, or a separate circuit, among other configurations.In regard to EAC-VRSC 1202, once unknown audio MMAU 1120 is received atEAC-VRS NW processor (processor) 1204, delay 1504 delays the audio thatis sent to speaker 1114 a (MMA_(K) 1118′), and un-delayed audio MMA_(K)1118 is sent to AEC 1306 for processing with audio received by MEMs mic1306 a-n. In regard to video and audio signals received via HDMI audioextractor 1206, according to aspects of the embodiment, an audio signaland its accompanying video signal can both be delayed a similar amount,then routed to video display 1134, and sound bar 1132. According tofurther aspects of the embodiments, the video signal does not need to bedelayed, especially if the delay is kept very low, e.g., on the order ofabout 1-2 milliseconds.

As briefly discussed above, FIG. 16 illustrates the timing of thedifferent signals and their respective processing. Although FIG. 16illustrates the audio signals as discrete waveforms, those of skill inthe art will understand that that is generally not the case, but hasbeen done merely to make the illustration easier to understand.Nonetheless, the illustration of relative timing between the variousaudio signal and processing is the same regardless of whether thesignals are discrete or substantially continuous in time.

Attention is now directed to FIG. 17, which illustrates a simplifiedblock diagram showing the principle of determining an appropriate timedelay between the broadcasting of known audio from one or more speakersand receipt by a microphone associated with an acoustic echocancellation device wherein knowledge of the delay increases theefficacy of echo cancellation and other audio system processes accordingto aspects of the embodiments.

As those of skill in the art can appreciate, however, there aredifferent methods in which to implement a delay of transmitted andbroadcast audio signals. As described above, a physical delay device(delay 1504) can be added in one or more circuit locations. Anothermeans for incorporating delay in a broadcast audio signal is to maintaina known physical separation between the speakers and microphones. FIG.17 illustrates how each of these delays can be determined during systemdesign. In FIG. 17, the physical delay device, delay D1, or delay 1504,is located on an output of audio received by system 1200. Then, as shownin FIG. 17, the time it takes for the audio to get from the source(within system 1200) to speakers 1114/1132 is designed at time t2; thetime through delay D1 and then output through speakers 1114/1132, andthen when received by MEMs mic 1306.

Delay D2 is the time it takes for sound to arrive at MEMs mic 1306 afterbeing broadcast by speakers 1114/1132. That is, delay D2 is determinedby a separation distance between speakers 1114/1132 and MEMs mic 1306,as shown in FIG. 17, which is Δx. According to aspects of theembodiments, it has been determined that a delay of about 2 ms isappropriate in order for AEC circuit 1306 to have time to processpreviously unknown audio MMA_(U) 1120 into known audio MMA_(K) 1118. Adelay of 2 ms from speaker 1114 to mic 1116 indicates a speaker-micseparation distance of about 0.686 meters (at standard temperature,pressure, and at sea level), which is about 2.25 feet, or 2′3″.

Referring back to FIG. 13, once the combination of known audio MMA_(K)1118, delayed known audio MMA_(K) 1118′, and unknown voice audio 1124 isreceived by MEMs mic 1306, and processed by AEC circuit 1306 (whichperforms AEC and other audio signal processing techniques), the nowprocessed audio, which comprises substantially only of spoken words, isfirst processed to determine if the keyword is present. As brieflydescribed above, keyword detection is performed locally for severalreasons. A first reason is privacy; if the speaker or speakers 1122 donot intend to use EAC-VRS 1200 according to aspects of the embodiments,then there is a reasonable expectation that their conversation isconsidered private, and should not be transmitted to the cloud or VRSv1110. While every attempt is made to keep VRSv 1110 and EAC-VRS 1200secure from hackers, it is not always possible to guarantee suchprotection. In addition, transferring large audio files that are notgoing to be processed can be a waste of valuable bandwidth throughinternet 1126 and use of VRSv 1110.

Thus, keyword detection occurs locally according to aspects of theembodiments. Such keyword detection can occur in any of AEC circuit1304, DDB circuit 1302, or transceiver 1208; once detection of thekeyword has occurred, the remainder of the conversation is digitized andtransmitted to VRSv 1110, in a manner known to those of skill in theart. A discussion of the processing that occurs from this point onwardsis both not needed to understand the aspects of the embodiments, norwithin the scope of this discussion and as such has been omitted infulfillment of the dual purposes of clarity and brevity.

Transmission of the large audio data files can be transmitted vianetwork connection 1106 for voice recognition, intercom, and otherpurposes. Network 1126 can include the cloud, the internet, a LAN, WAN,and other types of networks. Various communications and networkprotocols can be used, such as session initiation protocol (SIP), which,as those of skill in the art can appreciate, is one of the most commonprotocols used in VoIP technology. SIP is an application layer protocolthat works in conjunction with other application layer protocols tocontrol multimedia communication sessions over the Internet. Other typesof low latency network protocols include DANTE, AES-67 or AVB, amongothers. AES67 is a technical standard for audio over IP, and audio overEthernet interoperability. As those of skill in the art can appreciate,the AES67 standard was developed by the Audio Engineering Society andfirst published in September 2013. It is a layer 3 protocol suite basedon existing standards and is designed to allow interoperability betweenvarious IP-based audio networking systems such as RAVENNA, Livewire,Q-LAN and Dante. It also provides interoperability with layer 2technologies, like Audio Video Bridging (AVB). AVB is an audio-videobased network that implements a set of protocols developed by the IEEE802.1 Audio/Video Bridging Task Group. AVB works by reserving a fractionof the available Ethernet bandwidth for AVB traffic. There are other lowlatency wireless protocols such as Bluetooth, 802.11, among others, thatcan be used, including proprietary protocols. A wired network interfacecan also be used for audio and configuration communication, as well asfor power, using PoE.

There are numerous technologies and devices that currently exist thatcan be used in physical implementations of the aspects of theembodiments. By way of non-limiting example, one physical configurationthat can be used implements four pulse density modulation (PDM) MEM mics1306 connected to an XMOS manufactured “Far Field Voice Processor,”acting as EAC VRS 1204. For the networking aspect of the implementation,a Dante Ultimo integrated circuit (IC), manufactured by Audinate, can beused for audio networking. The Dante Ultimo network IC providesbidirectional pulse code modulation (PCM) audio communication. Onedirection would be the microphone output, and the other would be the AECreference input. The XMOS device can be connected to the Ultimo deviceusing an I²S interface. As those of skill in the art can appreciate, theI²S interface is an electrical serial bus interface standard used forconnecting digital audio devices together. It can be used to communicatePCM audio data between integrated circuits in an electronic device. TheI²S bus separates clock and serial data signals, resulting in low jitterrates than is typical of communications systems that recover the clockfrom the data stream.

As briefly described above, according to further aspects of theembodiments, audio from video and external audio signals can also beextracted, and used as a reference signal in AEC circuit 1304. Referringback to FIG. 12, external video and audio signals are transmitted to EACVRS 1200, and is received by audio extractor 1206. Audio extractor 1206extracts the audio signal—referred to as MMA_(U) 1120—from the videosignal—making it into MMA_(K) 1118, known audio, for use in AECprocessing, as described above. MMA_(K) 1118 is sent to NW processor1204. as shown in FIG. 15, delay 1504 delays both the audio and videosignal substantially equal amount. The delay provides for properprocessing as discussed above. According to further aspects of theembodiments, delay 1504 does not necessarily have to delay the video assmall audio delays are generally not noticeable, but can be included tokeep the audio and video in as good a sync situation as possible. Use ofdelay 1504 provides that audio sent to AEC circuit 1304 is receivedahead of the audio broadcast by speakers 1114/1132 and received by MEMsmic 1306. According to further aspects of the embodiments, HDMI videointerface 1206 can also perform de-interlacing and/or scaling. Inaddition, as discussed above, low latency network connections cantransmit the known audio reference signal, MMA_(K) 1118 to AEC circuit1306 and mic 1116. Such networks include DANTE and AES-67, with fixedand configurable delays in the range of a couple of milliseconds.

According to still further aspects of the embodiments, HDMI videointerface 1206 can be made as two different components. That is, in afirst aspect of the embodiments, HDMI video interface 1206 can bemanufactured to be part of, or attached to HDMI Transceiver 1128. HDMITransceiver 1128 can have a wired or wireless output that can includethe output of HDMI video interface 1206, all of the outputs of which canbe wired or wireless. According to still further aspects of theembodiments, a simple video and audio transmitter can be located at theoutput of HDMI Transceiver 1128, and the HDMI video and audio signal canbe transmitted to audio sound bar 1132, video display 1134, either bywired or wireless means, and also transmitted to NMD 1212; in this case,delay 1504 can be located in NMD 1212 (recalling that small differencesin timing between audio and video can be inconsequential when the videois viewed on a monitor). In addition, delay 1504 can also be located inaudio amplifier 1210, and all of these spatially separated delays 1504can be adapted to receive a substantially similar delay configurationsignal that proscribes a fixed amount of delay for the audio signal,regardless of its source.

Referring back to FIG. 13, there is also shown optional networkinterface device (NID) 1308 (which can also be implemented in theconfiguration of FIG. 14, according to aspects of the embodiments,although the same has not been included in FIG. 14, in fulfillment ofthe dual purposes of clarity and brevity; its operation in theconfiguration of FIG. 14 is substantially similar to that of FIG. 13 andthus additional discussion is not needed). According to aspects of theembodiments, the audio output signal from MEMs mic 1306 may need to beconverted to a format useful for cloud processing or VoIP calls. NID1308 can be located within each of MEMs mic 1306, within AEC circuit1304, or within DDB circuit 1302, or NID 1310 can be a stand-alonedevice, as shown in FIG. 13. If located in as a stand-alone device, NID1308 can convert between the low latency real time audio networkprotocol and the higher-level protocols like hyper text transportprotocol (http), session initiation protocol (SIP), or real timeprotocol (RTP). According to further aspects of the embodiments, it isalso possible that a multitude of NMDs 1212 can be used throughout theroom to improve coverage and improve SNR. In that case, NID 1308 can beused to combine the outputs of the plurality of NMDs 1212 using variousalgorithms (e.g., auto-mixing) to create a single voice output of higherquality.

FIG. 18 illustrates a block diagram view of external audio compensatedvoice recognition system with a surround sound stereo system (EAC-VRS-S)1800 according to further aspects of the embodiment. EAC-VRS-S 1800incorporates many of the same components as system 1200 shown in FIG.12, though many have been eliminated from FIG. 18 in order to make thefigure clearer and easier to understand. Nonetheless, those of skill inthe art can or should appreciate that EAC-VRS-S 1800 operatessubstantially similarly to that of system 1200, and includes all of thesame components. According to aspects of the embodiments, EAC-VRS-S 1800comprises surround sound stereo source (stereo source) 1802, surroundsound audio transfer cable (audio cable) 1804, surround sound speakers1806 a-h, surround sound audio transfer interface (audio interface)1808, and surround sound audio transfer down-mixer (audio down-mixerdevice) 1810.

In EAC-VRS-S 1800, a 7.1 stereo surround system is included and operatesin a conventional manner. In order to compensate for the complex audiooutput from the eight speakers when recovering unknown voice audio 1124,however, according to aspects of the embodiments, the complex 7.1 stereoaudio signal is down-mixed to a mono or “true” stereo stream, which isthen used as the un-delayed reference signal, MMA_(K) 1118, and input toAEC circuits 1304 in NMD 1112. As shown in FIG. 18, the complex 7.1stereo audio signal is transmitted first to audio interface 1808, whichthen passes the complex 7.1 stereo audio signal to both down-mixercircuit 2002 (shown in FIG. 20, which is a detailed block diagram ofdown-mixer device 1810) and delay 1504. The output of delay 1504 is thensent to speakers 1806 a-h (1806 a—right rear; 1806 b, right middle; 1806c—right front; 1806 d—center front; 1806 e—left front; 1806 f—leftcenter; 1806 g—left rear; and 1806 h—subwoofer) to be broadcast viaaudio cable 1804 as delayed MMA_(K) 1118′. At the same time, the complex7.1 stereo audio signal is also input to down-mixer circuit 2002.Down-mixer circuit 2002 generates an un-delayed mono or “true” stereoaudio signal, that is sent to EAC-VRS NW processor 1204 as un-delayedMMA_(K) 1118, which directs it to NMD 1212 via mic interface 1214.Un-delayed MMAK 1118 representing the down-mixed complex 7.1 stereoaudio signal is then forwarded to each AEC circuit 1304 in NMD 1212, tobe used in the manner as described in detail above to obtain an asaccurate as possible representation of unknown voice audio 1124according to aspects of the embodiments. According to further aspects ofthe embodiments, once the keyword has been determined, delay 1504 can beinstructed to mute its output i.e., shut down playing of the complex 7.1stereo audio signal so that the balance of the speech command can bemore readily understood.

According to further aspects of the embodiments, and referring back toFIGS. 12-20, any one or more of the paths that carry known audio,whether delayed or un-delayed, can be encrypted. In particular,according to still further aspects of the embodiments, un-delayed knownaudio MMAK 1118 can be encrypted prior to being sent to NMD 1212, andthe returned audio stream be encrypted as well as it is sent to EAC-VRSC1202 and then to VRSv 1110. That is, according to aspects of theembodiments, the output of DDB 1302 can be encrypted prior to be outputand transmitted to EAC-VRS NW processor 1204. In addition, and accordingto further aspects of the embodiments, known undelayed audio signalMMA_(K) 1118 can be encrypted by EAC-VRS NW processor 1302 prior to betransmitted to NMD 1212 from microphone interface 1214. Such encryptionprovides for additional security when EAC-VRS 1200 is used in acorporate board room, or meeting-room, setting, by way of non-limitingexample.

FIG. 21 illustrates a flow chart of method 2100 for compensating forknown audio in regard to unknown audio, so that the unknown audio canmore be more clearly determined using an external audio compensatedvoice recognition system (EAC-VRS) according to aspects of theembodiments. Method 2100 begins with method step 2102 in which externalaudio is received by EAC-VRS 1200; as described above, such externalaudio can be audio associated with an HDMI transmission, audio from VRSv1110, audio from an VoIP telephone call, or music from an externalstereo audio system, such as MMAS 1138.

In method step 2104, the received audio is transmitted to AEC 1304,which is located in NMD 1212. The transmitted audio can also be referredto as undelayed audio, as no delay is applied to it as the audio that isforwarded to one or more of speakers 1114 in method step 2106. Even ifthe audio is extracted from a combined video-audio signal, it isdelayed, and, according to aspects of the embodiments, the video can beequally delayed as well.

In method step 2108, which follows method step 2106, method 2100receives an output from each mic in NMD 1212. The output of each of themic's is the delayed version of the external audio received in methodstep 2102; the delay is inserted to that the AECs 1304 can have time toanalyze the externally received audio signal, and be ready to performprocessing when the same audio—at least in part—is received by the mics.That is, both the external audio and unknown voice audio is received ateach mic, and output to a respective AEC circuit 1304; then in methodstep 2108, the undelayed external audio is subtracted from the micoutput and what remains is substantially clearer unknown voice audio.

In decision step 2110, the unknown audio is checked to determine if thekeyword has been spoken to initiate the voice recognition process. Asthose of skill in the art can appreciate, the voice recognition processis typically used in a query-answer mode, in which the speaker speaksthe keyword, then asks a question, such as—“Keyword, can you tell mewhere to find a lobster?” If the question is ambiguous, or notrecognizable, voice recognition algorithms can query the speaker foradditional information. Or, if the question in not ambiguous, algorithmsin VRSv 1110 find a most suitable answer and return it to EAC-VRS 1202to be broadcast to the speaker. While such explanation is greatlysimplified, those of skill in the art can appreciate that a detaileddiscussion of how such voice recognition application operate is notneeded to understand the aspects of the embodiments, and therefore, infulfillment of the dual purposes of clarity and brevity, and moredetailed discussion has been omitted from herein.

If the keyword has not been received, and/or not understood (“No” pathfrom decision step 2110), method 2100 returns to step 2102 and continuesto receive external audio and perform method steps 2104-2110, until thecase when the method step is both received and understood (“Yes” pathfrom decision step 2110). Then, method 2100 proceeds to method step2112, wherein the process for transmitting the query to VRSv 1110begins. As those of skill in the art can appreciate, while the term“query” has been used herein, the spoken audio by user 1122 does notnecessarily need to be an actual question for it to be transmitted toVRSv 1110. Method 2100 then continues to transmit the audio spoken byuser 1122 (all the while continuously performing the receipt andsubtraction process as described in method steps 2102-2108) until eitherthe receipt of a transmission termination command (e.g., “stop”), or aperiod of silence that is at least as long as a predetermined period ofsilence that indicates an end of a query to VRSv 1110 (method step2114). Then, following the termination of the transmission, method 2100reverts back to method step 2102, and begins the process again. As thoseof skill in the art can appreciate, there is often the difficulty toillustrate graphically what occurs in a method in which substantiallycontinuous processing occurs; as such, the devices illustrated in FIGS.12-20 substantially continuously perform the processing stepsillustrated in FIG. 21 as embodied as method 2100 according to aspectsof the embodiments.

The embodiments discussed herein can also be embodied ascomputer-readable codes on a computer-readable medium. Thecomputer-readable medium can include a computer-readable recordingmedium and a computer-readable transmission medium. Thecomputer-readable recording medium is any data storage device that canstore data which can be thereafter read by a computer system. Examplesof the computer-readable recording medium include ROM, RAM, CD-ROMs andgenerally optical data storage devices, magnetic tapes, flash drives,and floppy disks. The computer-readable recording medium can also bedistributed over network coupled computer systems so that thecomputer-readable code is stored and executed in a distributed fashion.The computer-readable transmission medium can transmit carrier waves orsignals (e.g., wired or wireless data transmission through theInternet). Also, functional programs, codes, and code segments to, whenimplemented in suitable electronic hardware, accomplish or supportexercising certain elements of the appended claims can be readilyconstrued by programmers skilled in the art to which the embodimentspertain.

INDUSTRIAL APPLICABILITY

To solve the aforementioned problems, the aspects of the embodiments aredirected towards systems, methods, and modes for controllingcontrollable devices in the control network based on audio commandsalone, according to an aspect of the embodiments, and in further aspectsof the embodiments, controlling the controllable devices of the controlnetwork based on audio commands and other sensory information.

The disclosed embodiments provide a system, software, and a method fordetermining which of one or more controllable devices an audible commandis directed towards using one or more of speech recognition, time-datestamping, amplitude analysis, and other techniques, as described herein.It should be understood that this description is not intended to limitthe embodiments. On the contrary, the embodiments are intended to coveralternatives, modifications, and equivalents, which are included in thespirit and scope of the embodiments as defined by the appended claims.Further, in the detailed description of the embodiments, numerousspecific details are set forth to provide a comprehensive understandingof the claimed embodiments. However, one skilled in the art wouldunderstand that various embodiments may be practiced without suchspecific details.

Although the features and elements of aspects of the embodiments aredescribed being in particular combinations, each feature or element canbe used alone, without the other features and elements of theembodiments, or in various combinations with or without other featuresand elements disclosed herein.

This written description uses examples of the subject matter disclosedto enable any person skilled in the art to practice the same, includingmaking and using any devices or systems and performing any incorporatedmethods. The patentable scope of the subject matter is defined by theclaims, and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims.

The above-described embodiments are intended to be illustrative in allrespects, rather than restrictive, of the embodiments. Thus theembodiments are capable of many variations in detailed implementationthat can be derived from the description contained herein by a personskilled in the art. No element, act, or instruction used in thedescription of the present application should be construed as criticalor essential to the embodiments unless explicitly described as such.Also, as used herein, the article “a” is intended to include one or moreitems.

All United States patents and applications, foreign patents, andpublications discussed above are hereby incorporated herein by referencein their entireties.

ALTERNATE EMBODIMENTS

Alternate embodiments may be devised without departing from the spiritor the scope of the different aspects of the embodiments.

What is claimed is:
 1. A microphone system for determining one or more spoken words (microphone system), comprising: a receiving device adapted to receive an electrical audio signal from an external source (un-delayed external source audio signal); a delay device adapted to delay the un-delayed external source audio signal with a first predetermined delay (delayed external source audio signal); a first transmitter adapted to transmit the delayed external source audio signal to one or more loudspeakers to generate a delayed acoustic external source audio signal (undesired acoustic audio signal); a second transmitter adapted to transmit the un-delayed external source audio signal to a first input of an acoustic echo cancellation (AEC) device as a reference audio signal; one or more microphones within the microphone system adapted to receive desired acoustic audio signals generated by one or more people and to receive the undesired acoustic audio signals, and wherein the one or more microphones are further adapted to convert the received signals from acoustical energy signals into electrical audio signals and outputting them as microphone output audio signals; and an acoustic echo cancellation (AEC) device adapted to receive the microphone output audio signals at a second input of the AEC device, and wherein the AEC device is further adapted to receive the reference audio signal at a first input of the AEC device, and wherein the AEC device is further adapted to process the reference audio signal and the microphone output audio signals in the AEC device, wherein the first predetermined delay is substantially equal to a second delay that occurs between the generation of the undesired acoustic audio signals by the one or more loudspeakers and the processing of the microphone output audio signals, and wherein the AEC device is adapted to cancel substantially all of the reference audio signal from each of the microphone output audio signals in the AEC device, and further wherein the AEC device is adapted to output the same from the AEC as a corrected audio signal.
 2. The microphone system according to claim 1, wherein the AEC device is further adapted to subtract the reference audio signal from each of the microphone output audio signals when cancelling, and outputting the result as the corrected audio signal.
 3. The microphone system according to claim 1, further comprising: a direction detection and beamforming (DDB) circuit adapted to receive two or more corrected audio signals and combine the same into a single audio output signal.
 4. The microphone system according to claim 3, wherein the DDB is further adapted to encrypt the corrected audio signal prior to being output from the DDB.
 5. The microphone system according to claim 1, wherein the external speakers are located within an audible detection radius of the AEC circuit.
 6. The microphone system according to claim 1, wherein the external speakers are located within hearing distance of the two or more microphones.
 7. The microphone system according to claim 1, further comprising: a keyword recognition device adapted to detect the presence of a spoken keyword in the corrected audio signal.
 8. The microphone system according to claim 7, wherein the microphone system is further adapted to initiate transmission of the corrected audio signal to a voice recognition server through a network following detection of the spoken keyword, and to terminate transmission of the corrected audio signal upon the occurrence of a termination event.
 9. The microphone system according to claim 8, wherein the termination event comprises: at least one of a timeout condition, and one or more termination words.
 10. The microphone system according to claim 1, wherein the corrected audio signal comprises substantially only the desired spoken audio signal.
 11. The microphone system according to claim 1, further comprising: a voice recognition system adapted to receive and perform a speech recognition analysis on the corrected audio signal.
 12. The microphone system according to claim 11, wherein the voice recognition system is further adapted to respond to the recognized corrected audio signal.
 13. The microphone system according to claim 1 wherein the microphone system is adapted to be used in a full duplex intercom mode.
 14. The microphone system according to claim 1, wherein the microphone system is adapted to enable full duplex intercom conversations between a first microphone system and one or more additional microphones.
 15. The microphone system according to claim 1 wherein the microphone system is adapted to be used as a telephone system.
 16. The microphone system according to claim 15, wherein the microphone system is adapted to enable telephone conversations between a first microphone and one or more additional microphones.
 17. The microphone system according to claim 1, wherein the microphone comprises: an Ethernet network device.
 18. The microphone system according to claim 17, wherein the Ethernet network device is adapted to receive power over a power-over-Ethernet interface.
 19. The microphone system according to claim 1, wherein the network is associated with a voice recognition system.
 20. The microphone system according to claim 1, wherein the reference audio signal is encrypted prior to being transmitted from the network to the AEC device.
 21. A microphone system, comprising: one or more microphones each of which is adapted to receive acoustic audio signals and convert the same from acoustical energy signals into electrical audio signals and output the same as microphone output audio signals; an acoustic echo cancellation (AEC) device adapted to receive the microphone output audio signals, and wherein the AEC device also receives as a second input a reference audio signal from a network, and wherein the AEC device is further adapted to subtract the reference audio signal from the microphone output audio signals, and output the same as a corrected audio signal, and wherein the reference audio signal comprises an audio signal generated by an external audio system, and further wherein the received acoustic audio signals comprises a combination of a desired spoken audio signal and a delayed version of undesired audio signals, and wherein the reference audio signal comprises an undelayed version of the undesired audio signals, and further wherein the corrected audio signal comprises substantially only the desired spoken audio signal; and a delay circuit adapted to delay substantially all of the undesired audio signals prior to the undesired audio signals being broadcast by one or more speakers within hearing distance of the microphone system.
 22. The microphone system according to claim 21, further comprising: a direction detection and beamforming (DDB) circuit adapted to receive two or more corrected audio signals and combine the same into a single audio output signal.
 23. The microphone system according to claim 22, wherein the DDB is further adapted to encrypt the corrected audio signal prior to being output from the DDB. 